# Get ucharclasses transitions right for Unicode symbols

I am trying to come up with a universal-ish boilerplate that will let me use as many weird Unicode symbols as I like in my plaintext documents, and handle font fallback on those symbols gracefully when exporting to PDF (using Pandoc with XeLaTeX). Here is a sample document:

Some arrows such as ←, ⇔, ↗. Is the main font back on?

Some symbols such as 🤷, 🤦, 🎉, 🎊, 📧, 📱. Is the main font back on?

Some mathematical operators such as ≥, ≡, ≈. Is the main font back on?

Some letters and numerals such as 𝒞, 𝔸, 𝔽 and Ⅴ. Is the main font back on?

Quid des caractères accentués ? Is the main font back on?


Here is what I expect from the boilerplate:

• fit in a file I can include with pandoc's --include-in-header
• switch between two fonts: the "main" one for Latin/punctuation, the "default" one for symbols

So far, here is my most successful attempt*:

\usepackage{fontspec}
\usepackage[Latin,Mathematics,Punctuation,Symbols]{ucharclasses}

\newfontfamily{\mydefaultfont}{Symbola}
\newfontfamily{\mymainfont}{DejaVu Sans}

\setTransitionsForPunctuation{\mymainfont}{\mydefaultfont}
\setTransitionsForLatin{\mymainfont}{\mydefaultfont}
\setTransitionsForSymbols{\mydefaultfont}{\mymainfont}
\setTransitionsForMathematics{\mydefaultfont}{\mymainfont}


(NB: I run Pandoc with -V mainfont="DejaVu Sans", which sticks \setmainfont[]{DejaVu Sans} somewhere in the TeX source)

I say "most successful", because I still get "glitches" here and there:

Specifically:

• commas after 🤷 (U+1F937 SHRUG) and 🤦 (U+1F926 FACE PALM) are displayed with Symbola rather than DejaVu Sans;
• Latin letters following accented characters are displayed with Symbola rather than DejaVu Sans.

In addition to the obvious question ("How do I get my boilerplate to do what it's supposed to?"), I would like to add another one: is there an "easy" way** to get the name of the block a character belongs to?

* This is the result of several iterations, in which I tried to solve a bunch of issues such as:

• XeLaTeX silently dropping some characters
• DejaVu Sans trying to display characters it does not have (yielding white boxes)

** I.e. as automated as possible; I figured I could write a C program to get the answer from libicu, but the library only defines an enumeration without human-readable strings. The "cleanest" approach I could find consisted in downloading and parsing http://unicode.org/Public/UNIDATA/Blocks.txt, which is not exactly straightforward. So far I am left with "Ask fileformat.info".

Environment:

• Debian Jessie
• XeTeX 3.14159265-2.6-0.99992 (TeX Live 2015/dev/Debian)
• ucharclasses 2012/09/25 v2.0x

NB: I have just noticed that ucharclasses only supports Unicode up to version 8.0. SHRUG and FACE PALM appear to have been introduced by Unicode 9.0 (according to fileformat.info). I guess this is tied to the commas not reverting to DejaVu Sans, but how exactly? And what can be done about this?

EDIT: Replacing the \setTransitionsForPunctuation and \setTransitionsForLatin to \setTransitionTo{XXX} solves the issue following accented characters, but causes DejaVu Sans to be used for SHRUG and FACE PALM (yielding white boxes). I am suspecting an ucharclasses issue when transitionning between one subset of an informal group to another (here, from LatinSupplement, which contains "è", to regular Latin).

OK, so SHRUG and FACE PALM appear to have been added to Unicode 9's Supplemental Symbols and Pictographs block. The latest version of ucharclasses does support this block; I chose to patch my local installation like so:

@@ -212,6 +212,8 @@
\do{TransportAndMapSymbols}{128640}{128767}
\do{AlchemicalSymbols}{128768}{128895}
\do{CJKUnifiedIdeographsExtensionD}{177984}{178207}
+  \do{SupplementalSymbolsAndPictographs}{129280}{129535}
}
% ----------------------------------------------------------------------------
%  Option handling lets the user turn off "load all" and selectively enable only those blocks
@@ -396,6 +398,7 @@
\do{Emoticons}
\do{TransportAndMapSymbols}
\do{AlchemicalSymbols}
+  \do{SupplementalSymbolsAndPictographs}
}

\def\YiClasses{


I also had to tweak my header file; here is the final version:

\usepackage[Latin,Mathematics,NumberForms,Punctuation,Symbols]{ucharclasses}

\newfontfamily{\mydefaultfont}{Symbola}
\newfontfamily{\mymainfont}{DejaVu Sans}

\setTransitionsForSymbols{\mydefaultfont}{\mymainfont}
\setTransitionsFor{NumberForms}{\mydefaultfont}{\mymainfont}
\setTransitionsForMathematics{\mydefaultfont}{\mymainfont}
\setTransitionTo{Punctuation}{\mymainfont}
\setTransitionTo{Latin}{\mymainfont}


Comment is too small, so posting as answer, and not about ucharclasses at all, just unicode.

===

Unicode is huge.

as many weird Unicode symbols as I like -- the magnitude of "many" may be limited by system constraints.

Using an empty edit window, pasting in a string of characters (one character from each of the first 120 unicode blocks), takes >5 min, <10min to complete (I went away and did something else). Compiling is OK (with one font):

Pasting in the next 110 crashed the editor (Texworks in Miktek on Windows).

Similarly, 3 min to open the tex file later. I expect it's buffer-related in some way.

In the other dimension, there seems to be a threshold limit of about 60 font families definable per session, after which Xelatex fails.

But all operational matters are fixable.

MWE

\documentclass[12pt]{article}
\usepackage[no-math]{fontspec}
\setmainfont{Code2003}

\begin{document}
abc All of this is one font -- Code2003: The first ~120 unicode blocks (with some scattered spaces to activate line-wrapping):

\Large
!¡Āƀɐʰ́ ͰБԀԱאعݐބߒࠅࡁࢢऄঅਅઅଅஅ అಕഅඅกກ༂က  ႠᄁሁᎀᎠ ᐂᚄᚠᜀᜠᝀᝠក᠀ᢰᤀᥐᦀ ᧠ᨀ ᨢ᪰ᬒᮃᯂᰂ᱒ᲀ᳀᳑ᴁᶀ᷁ḁἁ\-•⁵₤⃔ℂ ⅒→∂⌂␂⑂①━▂▢☀✅⟁⟱ ⠗⤃⦃⨂⬂ⰂⱢ Ⲁⴂⴳⶁⷡ ⸙⺒⼒⿲〖あアㄅㄳ㆕ㆣ ㇃ㇲ ㈃㌂㐂䷂万ꀂ꒓ꔂꙂꚣ꜃Ꜳꠃꡂꢃꢑ

% ꣴ꤁ꤰꥠꦅꧡꨁꩤꪃꫢꬁꬱꭱꯀ가ힰ更ﬃﭒ︙︻︴ﺃ＃𐀁𐂂𐄓𐅒𐆒𐇐𐊃𐊣𐋢𐌇𐌸𐍒𐎄𐎠𐐂𐑔𐒄𐒲𐔀𐔰𐘷𐠃𐡄𐡣𐢃𐣣𐤄𐤤𐦄𐦥𐨒𐩣𐪀𐫂𐬃𐭄𐭤𐰇𐲃𐹢𑀆𑃑𑅓𑆀𑇤𑈃𑊳𑌃𑐂𑒂𑖂𑘂𑙢𑚃𑢤𑫄𑰄𑱳𒀣𒐃𒒂𔐂𖠂𖩂𖫒𖬂𖼃𗀂𘠂𛰂𝀄𝀔𝈃𝌃𝍣𝐄𝠃𞀃𞠂𞤄𞸑🀃🂥🂥🄆🌅😄🙓🚂🜃🞃🠄🤔𠀓𪜃𫝃𫠣再󠀤
\end{document}


For multi-block character matters and single words or so across many scripts, ucharclasses is useful. Or for a few scripts per document, for long texts (the general case, I imagine).

For multi-lingual and/or multi-script long-texts, polyglossia or babel, because of hyphenation, translations and other language-level and layout-related things (combined with biblatex/biber for bibliographies).