1/* Part of SWI-Prolog 2 3 Author: Jan Wielemaker 4 E-mail: J.Wielemaker@vu.nl 5 WWW: http://www.swi-prolog.org 6 Copyright (c) 2010-2026, University of Amsterdam 7 CWI, Amsterdam 8 SWI-Prolog Solutions b.v. 9 All rights reserved. 10 11 Redistribution and use in source and binary forms, with or without 12 modification, are permitted provided that the following conditions 13 are met: 14 15 1. Redistributions of source code must retain the above copyright 16 notice, this list of conditions and the following disclaimer. 17 18 2. Redistributions in binary form must reproduce the above copyright 19 notice, this list of conditions and the following disclaimer in 20 the documentation and/or other materials provided with the 21 distribution. 22 23 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 24 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 25 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 26 FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 27 COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 28 INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 29 BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 30 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 31 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 32 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 33 ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 34 POSSIBILITY OF SUCH DAMAGE. 35*/ 36 37:- module(unicode, 38 [ unicode_property/2, % ?Code, ?Property 39 unicode_map/3, % +In, -Out, +Options 40 unicode_nfd/2, % +In, -Out 41 unicode_nfc/2, % +In, -Out 42 unicode_nfkd/2, % +In, -Out 43 unicode_nfkc/2, % +In, -Out 44 unicode_nfkc_casefold/2, % +In, -Out 45 unicode_casefold/2, % +In, -Out 46 unicode_version/1, % -Version 47 unicode_codepoint_valid/1, % +Code 48 atom_graphemes/2, % ?Atom, ?Graphemes 49 string_graphemes/2 % ?String, ?Graphemes 50 ]). 51:- use_foreign_library(foreign(unicode4pl)).

Unicode string handling

This library wraps the utf8proc library, giving Prolog code access to Unicode character properties, string normalization, case folding, and grapheme-cluster iteration.

Three levels of API are provided:

Normalization: unicode_nfd/2, unicode_nfc/2, unicode_nfkd/2, unicode_nfkc/2 implement the four standard Unicode normalization forms (NFD, NFC, NFKD, NFKC; see UAX#15). unicode_nfkc_casefold/2 combines NFKC with case folding for caseless identifier matching (see UAX#31).
Per-codepoint properties: unicode_property/2 queries the Unicode property database (general category, bidi class, decomposition type, display width, case mappings, grapheme boundary class, ...).

Mixed string-level transformations: unicode_map/3 is the workhorse; it accepts a list of flags chosen from a fixed set and performs the corresponding composition of decompose / compose / strip / lump / case-fold / grapheme-boundary-mark operations in a single pass. unicode_casefold/2 is a convenience wrapper.

Grapheme clusters (user-perceived characters) can be iterated with atom_graphemes/2 and string_graphemes/2.

Loading this library also installs a Unicode NFC normalisation hook into the SWI-Prolog kernel. The kernel's unicode_atoms policy (Prolog flag, stream property and read_term/2,3 option) uses this hook for its nfc and error modes; without the library loaded those modes raise existence_error(hook, unicode_normalize). The kernel's quoted-write rule that force-quotes atoms containing combining marks is independent and works even without this library loaded.

Lump handling:

U+0020      <-- all space characters (general category Zs)
U+0027  '   <-- left/right single quotation mark U+2018..2019,
                modifier letter apostrophe U+02BC,
                modifier letter vertical line U+02C8
U+002D  -   <-- all dash characters (general category Pd),
                minus U+2212
U+002F  /   <-- fraction slash U+2044,
                division slash U+2215
U+003A  :   <-- ratio U+2236
U+003C  <   <-- single left-pointing angle quotation mark U+2039,
                left-pointing angle bracket U+2329,
                left angle bracket U+3008
U+003E  >   <-- single right-pointing angle quotation mark U+203A,
                right-pointing angle bracket U+232A,
                right angle bracket U+3009
U+005C  \   <-- set minus U+2216
U+005E  ^   <-- modifier letter up arrowhead U+02C4,
                modifier letter circumflex accent U+02C6,
                caret U+2038,
                up arrowhead U+2303
U+005F  _   <-- all connector characters (general category Pc),
                modifier letter low macron U+02CD
U+0060  `   <-- modifier letter grave accent U+02CB
U+007C  |   <-- divides U+2223
U+007E  ~   <-- tilde operator U+223C

@see http://www.unicode.org/reports/tr15/ (UAX#15 Normalization) @see http://www.unicode.org/reports/tr29/ (UAX#29 Grapheme Clusters) @see http://www.unicode.org/reports/tr31/ (UAX#31 Identifiers) @see https://github.com/JuliaStrings/utf8proc

315unicode_property(Code, Property) :- 316 nonvar(Code), nonvar(Property), 317 !, 318 '$unicode_property'(Code, Property, false). 319unicode_property(Code, Property) :- 320 nonvar(Code), 321 !, 322 property(Property), 323 '$unicode_property'(Code, Property, true). 324unicode_property(Code, Property) :- 325 var(Code), 326 !, 327 between(0, 0x10ffff, Code), 328 property(Property), 329 '$unicode_property'(Code, Property, true). 330 331% The third argument of `$unicode_property/3` is a `Silent` flag. 332% When `true`, the C layer returns `false` instead of raising 333% `domain_error(unicode_property, ...)` for a property name the 334% binding was compiled without (older libutf8proc versions lack 335% some fields). Enumeration modes set it to `true` so 336% property/1 clauses for absent features are silently skipped. 337 338property(category(_)). 339property(combining_class(_)). 340property(bidi_class(_)). 341property(bidi_mirrored(_)). 342property(decomp_type(_)). 343property(ignorable(_)). 344property(boundclass(_)). 345property(width(_)). 346property(ambiguous_width(_)). 347property(uppercase(_)). 348property(lowercase(_)). 349property(titlecase(_)). 350property(indic_conjunct_break(_)).

402 /******************************* 403 * SANDBOX * 404 *******************************/ 405 406:- multifile 407 sandbox:safe_primitive/1. 408 409sandbox:safe_primitive(unicode:unicode_property(_,_)). 410sandbox:safe_primitive(unicode:unicode_map(_,_,_)). 411sandbox:safe_primitive(unicode:unicode_property(_,_)). 412sandbox:safe_primitive(unicode:unicode_version(_)). 413sandbox:safe_primitive(unicode:unicode_codepoint_valid(_)). 414sandbox:safe_primitive(unicode:atom_graphemes(_,_)). 415sandbox:safe_primitive(unicode:string_graphemes(_,_))