# ucharclasses misbehaves with Spacing Modifier Letters and Combining Diacritical Marks

The ucharclasses package makes it possible to automatically switch fonts (or do other useful things) based on the Unicode block of characters. However, it seems to be having trouble resetting the font after spacing modifier letters and combining diacritical marks, when used with fontspec (in this case, with XeLaTeX).

I've found a few approaches towards solving this problem, but none of them seem to fix it entirely.

I've made minimum working examples based on the normal use and several proposals I've found for dealing with related issues. All of the examples attempt to set a serif font (DejaVu Serif) for IPA Extensions, Combining Diacritical Marks, and Spacing Modifier Letters, and use a sans-serif font (DejaVu Sans) for everything else. Both fonts contain glyphs for all the characters tested with.

1. This mwe uses the conventional approach to set the font for these

\documentclass{article}

\usepackage{polyglossia}
\setdefaultlanguage[variant=british]{english}
\usepackage[Latin, Phonetics, Diacritics, SpacingModifierLetters]{ucharclasses}
\usepackage{fontspec}
\defaultfontfeatures{Scale=MatchLowercase,Mapping=tex-text}
\newfontfeature{IPA}{+mgrk}
\setmainfont[IPA]{DejaVu Sans}
\newfontfamily\dejavuserif[IPA]{DejaVu Serif}

\setTransitionsFor{IPAExtensions}{\dejavuserif}{\normalfont}
\setTransitionsFor{CombiningDiacriticalMarks}{\dejavuserif}{\normalfont}
\setTransitionsFor{SpacingModifierLetters}{\dejavuserif}{\normalfont}

\begin{document}
\thispagestyle{empty}

thaaw [tʰɑɑɯ] [tɑɑɯ] [tʰɑ́ɑɯ] [tɑ́ɑɯ] thaaw

\end{document}


In the output of this example, all characters in a "word" after a spacing modifier letter (in this case [ʰ], U+02B0) or combining diacritical mark (in this case [◌́], U+301) are displayed in the default sans-serif font.

2. This mwe uses a slight modification of the conventional approach, based on this answer.

\documentclass{article}

\usepackage{polyglossia}
\setdefaultlanguage[variant=british]{english}
\usepackage[Latin, Phonetics, Diacritics, SpacingModifierLetters]{ucharclasses}
\usepackage{fontspec}
\defaultfontfeatures{Scale=MatchLowercase,Mapping=tex-text}
\newfontfeature{IPA}{+mgrk}
\setmainfont[IPA]{DejaVu Sans}
\newfontfamily\dejavuserif[IPA]{DejaVu Serif}

\setTransitionTo{IPAExtensions}{\dejavuserif}
\setTransitionFrom{IPAExtentions}{\normalfont}
\setTransitionTo{CombiningDiacriticalMarks}{\dejavuserif}
\setTransitionFrom{CombiningDiacriticalMarks}{\normalfont}
\setTransitionTo{SpacingModifierLetters}{\dejavuserif}
\setTransitionFrom{SpacingModifierLetters}{\normalfont}

\begin{document}
\thispagestyle{empty}

thaaw [tʰɑɑɯ] [tɑɑɯ] [tʰɑ́ɑɯ] [tɑ́ɑɯ] thaaw

\end{document}


I didn't expect this approach to be any different from the conventional one, but there is a small difference: after the one "word" without one of the problem characters (i.e., [tɑɑɯ]), all the characters ("] [t") until the next problem character (ʰ) are rendered using the serif font. Otherwise, it still renders the remaining characters after the problematic characters with the sans-serif font.

3. This mwe uses an approach presented in this answer.

\documentclass{article}

\usepackage{polyglossia}
\setdefaultlanguage[variant=british]{english}

\usepackage[Latin, Phonetics, Diacritics, SpacingModifierLetters]{ucharclasses}

\usepackage{fontspec}
\defaultfontfeatures{Scale=MatchLowercase,Mapping=tex-text}
\newfontfeature{IPA}{+mgrk}

\setmainfont[IPA]{DejaVu Sans}
\newfontfamily\dejavuserif[IPA]{DejaVu Serif}

\setTransitionsFor{IPAExtensions}{\begingroup\dejavuserif}{\endgroup}
\setTransitionsFor{CombiningDiacriticalMarks}{\begingroup\dejavuserif}{\endgroup}
\setTransitionsFor{SpacingModifierLetters}{\begingroup\dejavuserif}{\endgroup}

\begin{document}
\thispagestyle{empty}

thaaw [tʰɑɑɯ] [tɑɑɯ] [tʰɑ́ɑɯ] [tɑ́ɑɯ] thaaw

\end{document}


The output of this approach is slightly better than the previous ones, in that the Combining Diacritical Mark (◌́) behaves as expected—i.e., the characters after it are all rendered in the serif font, but only if there's not another problematic character before it (like ʰ). It also produces the following errors to the console:

! Extra \endgroup.
<XeTeXinterchartoks> \endgroup

l.23 tháaw [tʰɑɑɯ]
[tɑɑɯ] [tʰɑ́ɑɯ] [tɑ́ɑɯ] tháaw
?
! Extra \endgroup.
<XeTeXinterchartoks> \endgroup

l.23 thaaw [tʰɑɑɯ] [tɑɑɯ] [tʰɑ́ɑɯ]
[tɑ́ɑɯ] thaaw
?
[1] (./2016-04-02b.aux) )

4. This mwe uses the solution suggested at this answer.

\documentclass{article}

\usepackage{polyglossia}
\setdefaultlanguage[variant=british]{english}
\usepackage[Latin, Phonetics, Diacritics, SpacingModifierLetters]{ucharclasses}
\usepackage{fontspec}
\defaultfontfeatures{Scale=MatchLowercase,Mapping=tex-text}
\newfontfeature{IPA}{+mgrk}
\setmainfont[IPA]{DejaVu Sans}
\newfontfamily\dejavuserif[IPA]{DejaVu Serif}

\makeatletter
\setTransitionsFor{IPAExtensions}
{\let\curfamily\f@family\let\curshape\f@shape\let\curseries\f@series\dejavuserif}
{\fontfamily{\curfamily}\fontshape{\curshape}\fontseries{\curseries}\selectfont}
\setTransitionsFor{SpacingModifierLetters}
{\let\curfamily\f@family\let\curshape\f@shape\let\curseries\f@series\dejavuserif}
{\fontfamily{\curfamily}\fontshape{\curshape}\fontseries{\curseries}\selectfont}
\setTransitionsFor{CombiningDiacriticalMarks}
{\let\curfamily\f@family\let\curshape\f@shape\let\curseries\f@series\dejavuserif}
{\fontfamily{\curfamily}\fontshape{\curshape}\fontseries{\curseries}\selectfont}
\makeatother

\begin{document}
\thispagestyle{empty}

thaaw [tʰɑɑɯ] [tɑɑɯ] [tʰɑ́ɑɯ] [tɑ́ɑɯ] thaaw

\end{document}


The output of this approach seems to suggest that that the Combining Diacritical Mark (◌́) is not a problem—i.e., following characters in a word are displayed with the serif font if there isn't another character (like [ʰ]) making them sans serif—but the serif "spreads" into the next word as well.

I suspect this is a bug in or limitation of the ucharclasses package. Confirmation of this, or a solution or work-around would be quite welcome.

• One would have to make tests with the primitive \XeTeXcharclass etc to find out if it is a xetex or a ucharclass problem. But your character classes are not really disjoint. In your own example you are using "t" inside and outside the brackets. And what should happen with a combining accent used in normal text? A clear semantic markup is imho better. – Ulrike Fischer Apr 3 '16 at 11:50
• What do you mean that the classes are not really disjoint, @UlrikeFischer? I understand that I'm mixing blocks in words, but this is completely normal use of these characters, so I guess the behaviour should at least be predictable. Also, by "clear semantic markup", do you mean just defining a command for use with all phonetic transcriptions that sets the font (and anything else I might deem useful)? – Jonathan W. Apr 3 '16 at 15:44
• Also, my first response to your question would be that a combining diacritical mark in Latin text should probably be rendered in the serif font (given my definitions), but it makes me wonder whether combining marks and the character they're combining with can be different fonts (seems like a bad idea, even if possible?). – Jonathan W. Apr 3 '16 at 15:45