# Get ucharclasses transitions right for Unicode symbols

I am trying to come up with a universal-ish boilerplate that will let me use as many weird Unicode symbols as I like in my plaintext documents, and handle font fallback on those symbols gracefully when exporting to PDF (using Pandoc with XeLaTeX). Here is a sample document:

Some arrows such as ←, ⇔, ↗. Is the main font back on?

Some symbols such as 🤷, 🤦, 🎉, 🎊, 📧, 📱. Is the main font back on?

Some mathematical operators such as ≥, ≡, ≈. Is the main font back on?

Some letters and numerals such as 𝒞, 𝔸, 𝔽 and Ⅴ. Is the main font back on?

Quid des caractères accentués ? Is the main font back on?


Here is what I expect from the boilerplate:

• fit in a file I can include with pandoc's --include-in-header
• switch between two fonts: the "main" one for Latin/punctuation, the "default" one for symbols

So far, here is my most successful attempt*:

\usepackage{fontspec}
\usepackage[Latin,Mathematics,Punctuation,Symbols]{ucharclasses}

\newfontfamily{\mydefaultfont}{Symbola}
\newfontfamily{\mymainfont}{DejaVu Sans}

\setTransitionsForPunctuation{\mymainfont}{\mydefaultfont}
\setTransitionsForLatin{\mymainfont}{\mydefaultfont}
\setTransitionsForSymbols{\mydefaultfont}{\mymainfont}
\setTransitionsForMathematics{\mydefaultfont}{\mymainfont}


(NB: I run Pandoc with -V mainfont="DejaVu Sans", which sticks \setmainfont[]{DejaVu Sans} somewhere in the TeX source)

I say "most successful", because I still get "glitches" here and there:

Specifically:

• commas after 🤷 (U+1F937 SHRUG) and 🤦 (U+1F926 FACE PALM) are displayed with Symbola rather than DejaVu Sans;
• Latin letters following accented characters are displayed with Symbola rather than DejaVu Sans.

In addition to the obvious question ("How do I get my boilerplate to do what it's supposed to?"), I would like to add another one: is there an "easy" way** to get the name of the block a character belongs to?

* This is the result of several iterations, in which I tried to solve a bunch of issues such as:

• XeLaTeX silently dropping some characters
• DejaVu Sans trying to display characters it does not have (yielding white boxes)

** I.e. as automated as possible; I figured I could write a C program to get the answer from libicu, but the library only defines an enumeration without human-readable strings. The "cleanest" approach I could find consisted in downloading and parsing http://unicode.org/Public/UNIDATA/Blocks.txt, which is not exactly straightforward. So far I am left with "Ask fileformat.info".

Environment:

• Debian Jessie
• XeTeX 3.14159265-2.6-0.99992 (TeX Live 2015/dev/Debian)
• ucharclasses 2012/09/25 v2.0x

NB: I have just noticed that ucharclasses only supports Unicode up to version 8.0. SHRUG and FACE PALM appear to have been introduced by Unicode 9.0 (according to fileformat.info). I guess this is tied to the commas not reverting to DejaVu Sans, but how exactly? And what can be done about this?

EDIT: Replacing the \setTransitionsForPunctuation and \setTransitionsForLatin to \setTransitionTo{XXX} solves the issue following accented characters, but causes DejaVu Sans to be used for SHRUG and FACE PALM (yielding white boxes). I am suspecting an ucharclasses issue when transitionning between one subset of an informal group to another (here, from LatinSupplement, which contains "è", to regular Latin).

OK, so SHRUG and FACE PALM appear to have been added to Unicode 9's Supplemental Symbols and Pictographs block. The latest version of ucharclasses does support this block; I chose to patch my local installation like so:

@@ -212,6 +212,8 @@
\do{TransportAndMapSymbols}{128640}{128767}
\do{AlchemicalSymbols}{128768}{128895}
\do{CJKUnifiedIdeographsExtensionD}{177984}{178207}
+  \do{SupplementalSymbolsAndPictographs}{129280}{129535}
}
% ----------------------------------------------------------------------------
%  Option handling lets the user turn off "load all" and selectively enable only those blocks
@@ -396,6 +398,7 @@
\do{Emoticons}
\do{TransportAndMapSymbols}
\do{AlchemicalSymbols}
+  \do{SupplementalSymbolsAndPictographs}
}

\def\YiClasses{


I also had to tweak my header file; here is the final version:

\usepackage[Latin,Mathematics,NumberForms,Punctuation,Symbols]{ucharclasses}

\newfontfamily{\mydefaultfont}{Symbola}
\newfontfamily{\mymainfont}{DejaVu Sans}

\setTransitionsForSymbols{\mydefaultfont}{\mymainfont}
\setTransitionsFor{NumberForms}{\mydefaultfont}{\mymainfont}
\setTransitionsForMathematics{\mydefaultfont}{\mymainfont}
\setTransitionTo{Punctuation}{\mymainfont}
\setTransitionTo{Latin}{\mymainfont}