Get ucharclasses transitions right for Unicode symbols

Question

I am trying to come up with a universal-ish boilerplate that will let me use as many weird Unicode symbols as I like in my plaintext documents, and handle font fallback on those symbols gracefully when exporting to PDF (using Pandoc with XeLaTeX). Here is a sample document:

Some arrows such as ←, ⇔, ↗. Is the main font back on?

Some symbols such as 🤷, 🤦, 🎉, 🎊, 📧, 📱. Is the main font back on?

Some mathematical operators such as ≥, ≡, ≈. Is the main font back on?

Some letters and numerals such as 𝒞, 𝔸, 𝔽 and Ⅴ. Is the main font back on?

Quid des caractères accentués ? Is the main font back on?

Here is what I expect from the boilerplate:

• fit in a file I can include with pandoc's --include-in-header
• switch between two fonts: the "main" one for Latin/punctuation, the "default" one for symbols

So far, here is my most successful attempt*:

\usepackage{fontspec}
\usepackage[Latin,Mathematics,Punctuation,Symbols]{ucharclasses}

\newfontfamily{\mydefaultfont}{Symbola}
\newfontfamily{\mymainfont}{DejaVu Sans}

\setTransitionsForPunctuation{\mymainfont}{\mydefaultfont}
\setTransitionsForLatin{\mymainfont}{\mydefaultfont}
\setTransitionsForSymbols{\mydefaultfont}{\mymainfont}
\setTransitionsForMathematics{\mydefaultfont}{\mymainfont}

(NB: I run Pandoc with -V mainfont="DejaVu Sans", which sticks \setmainfont[]{DejaVu Sans} somewhere in the TeX source)

I say "most successful", because I still get "glitches" here and there:

Specifically:

• commas after 🤷 (U+1F937 SHRUG) and 🤦 (U+1F926 FACE PALM) are displayed with Symbola rather than DejaVu Sans;
• Latin letters following accented characters are displayed with Symbola rather than DejaVu Sans.

In addition to the obvious question ("How do I get my boilerplate to do what it's supposed to?"), I would like to add another one: is there an "easy" way** to get the name of the block a character belongs to?

* This is the result of several iterations, in which I tried to solve a bunch of issues such as:

• XeLaTeX silently dropping some characters
• DejaVu Sans trying to display characters it does not have (yielding white boxes)

** I.e. as automated as possible; I figured I could write a C program to get the answer from libicu, but the library only defines an enumeration without human-readable strings. The "cleanest" approach I could find consisted in downloading and parsing http://unicode.org/Public/UNIDATA/Blocks.txt, which is not exactly straightforward. So far I am left with "Ask fileformat.info".

Environment:

• Debian Jessie
• XeTeX 3.14159265-2.6-0.99992 (TeX Live 2015/dev/Debian)
• ucharclasses 2012/09/25 v2.0x

NB: I have just noticed that ucharclasses only supports Unicode up to version 8.0. SHRUG and FACE PALM appear to have been introduced by Unicode 9.0 (according to fileformat.info). I guess this is tied to the commas not reverting to DejaVu Sans, but how exactly? And what can be done about this?

EDIT: Replacing the \setTransitionsForPunctuation and \setTransitionsForLatin to \setTransitionTo{XXX} solves the issue following accented characters, but causes DejaVu Sans to be used for SHRUG and FACE PALM (yielding white boxes). I am suspecting an ucharclasses issue when transitionning between one subset of an informal group to another (here, from LatinSupplement, which contains "è", to regular Latin).

Answer 1

OK, so SHRUG and FACE PALM appear to have been added to Unicode 9's Supplemental Symbols and Pictographs block. The latest version of ucharclasses does support this block; I chose to patch my local installation like so:

@@ -212,6 +212,8 @@
   \do{TransportAndMapSymbols}{128640}{128767}
   \do{AlchemicalSymbols}{128768}{128895}
   \do{CJKUnifiedIdeographsExtensionD}{177984}{178207}
+  \do{SupplementalSymbolsAndPictographs}{129280}{129535}
 }
 % ----------------------------------------------------------------------------
 %  Option handling lets the user turn off "load all" and selectively enable only those blocks 
@@ -396,6 +398,7 @@
   \do{Emoticons}
   \do{TransportAndMapSymbols}
   \do{AlchemicalSymbols}
+  \do{SupplementalSymbolsAndPictographs}
 }

 \def\YiClasses{

I also had to tweak my header file; here is the final version:

\usepackage[Latin,Mathematics,NumberForms,Punctuation,Symbols]{ucharclasses}

\newfontfamily{\mydefaultfont}{Symbola}
\newfontfamily{\mymainfont}{DejaVu Sans}

\setTransitionsForSymbols{\mydefaultfont}{\mymainfont}
\setTransitionsFor{NumberForms}{\mydefaultfont}{\mymainfont}
\setTransitionsForMathematics{\mydefaultfont}{\mymainfont}
\setTransitionTo{Punctuation}{\mymainfont}
\setTransitionTo{Latin}{\mymainfont}

Answer 2

Comment is too small, so posting as answer, and not about ucharclasses at all, just unicode.

===

Unicode is huge.

as many weird Unicode symbols as I like -- the magnitude of "many" may be limited by system constraints.

Using an empty edit window, pasting in a string of characters (one character from each of the first 120 unicode blocks), takes >5 min, <10min to complete (I went away and did something else). Compiling is OK (with one font):

Pasting in the next 110 crashed the editor (Texworks in Miktek on Windows).

Similarly, 3 min to open the tex file later. I expect it's buffer-related in some way.

In the other dimension, there seems to be a threshold limit of about 60 font families definable per session, after which Xelatex fails.

But all operational matters are fixable.

MWE

\documentclass[12pt]{article}
\usepackage[no-math]{fontspec}
\setmainfont{Code2003}


\begin{document}
abc All of this is one font -- Code2003: The first ~120 unicode blocks (with some scattered spaces to activate line-wrapping): 

\Large
!¡Āƀɐʰ́ ͰБԀԱאعݐބߒࠅࡁࢢऄঅਅઅଅஅ అಕഅඅกກ༂က  ႠᄁሁᎀᎠ ᐂᚄᚠᜀᜠᝀᝠក᠀ᢰᤀᥐᦀ ᧠ᨀ ᨢ᪰ᬒᮃᯂᰂ᱒ᲀ᳀᳑ᴁᶀ᷁ḁἁ\-•⁵₤⃔ℂ ⅒→∂⌂␂⑂①━▂▢☀✅⟁⟱ ⠗⤃⦃⨂⬂ⰂⱢ Ⲁⴂⴳⶁⷡ ⸙⺒⼒⿲〖あアㄅㄳ㆕ㆣ ㇃ㇲ ㈃㌂㐂䷂万ꀂ꒓ꔂꙂꚣ꜃Ꜳꠃꡂꢃꢑ

% ꣴ꤁ꤰꥠꦅꧡꨁꩤꪃꫢꬁꬱꭱꯀ가ힰ更ffiﭒ︙︻︴ﺃ＃𐀁𐂂𐄓𐅒𐆒𐇐𐊃𐊣𐋢𐌇𐌸𐍒𐎄𐎠𐐂𐑔𐒄𐒲𐔀𐔰𐘷𐠃𐡄𐡣𐢃𐣣𐤄𐤤𐦄𐦥𐨒𐩣𐪀𐫂𐬃𐭄𐭤𐰇𐲃𐹢𑀆𑃑𑅓𑆀𑇤𑈃𑊳𑌃𑐂𑒂𑖂𑘂𑙢𑚃𑢤𑫄𑰄𑱳𒀣𒐃𒒂𔐂𖠂𖩂𖫒𖬂𖼃𗀂𘠂𛰂𝀄𝀔𝈃𝌃𝍣𝐄𝠃𞀃𞠂𞤄𞸑🀃🂥🂥🄆🌅😄🙓🚂🜃🞃🠄🤔𠀓𪜃𫝃𫠣再󠀤
\end{document}

For multi-block character matters and single words or so across many scripts, ucharclasses is useful. Or for a few scripts per document, for long texts (the general case, I imagine).

For multi-lingual and/or multi-script long-texts, polyglossia or babel, because of hyphenation, translations and other language-level and layout-related things (combined with biblatex/biber for bibliographies).