4

In (La)TeX you can define an active character that is only active in math mode by giving it a mathcode of "8000 and bringing a corresponding single-letter macro into scope. The same seems to work with Unicode characters in Xe(La)TeX:

\documentclass{article}
\usepackage{listings}

\begingroup
\catcode`φ=\active
\gdef φ{\varphi}
\endgroup
\mathcode`φ="8000

\begin{document}
\[ φ^φ \]
\end{document}

However, if we change \mathcode to a version using the XeTeX primitive \Umathcode

\Umathcode`φ="8 "0 "0

compilation breaks with Bad math class (8), though I think the syntax should be correct. The XeTeX documentation isn't very verbose on this topic, it doesn't even mention the case where the math class can be 8.

So what is the proper/recommended way to define such a character in Xe(La)TeX that is only active in math mode? Is using \mathcode fine or are there some subtle cases where it wouldn't work as expected?

1
  • Mathcode "8000 is very different from Umathcode "8 "0 "0. What makes you think they should be the same?
    – egreg
    Jun 20, 2017 at 21:25

2 Answers 2

5

A \Umathcode requires three numbers as argument:

  • the first number specifies the class and should be in the range 0-7;
  • the second number specifies the math family and should correspond to an allocated family;
  • the third number specifies the character slot.

In particular

\Umathcode`φ="8 "0 "0

is illegal.

It is similar to a \mathcode assignment, where the three pieces are packed together in a single hexadecimal number <class><family><slot> (two digits for the <slot>).

A \mathcode 32768 (hexadecimal "8000) denotes a math active character in standard TeX as well as in XeTeX. There is no difference at all under this respect.

I'd do the definition in a different way, but it's just a question of taste:

\begingroup\lccode`~=`φ \lowercase{\endgroup\def~}{\varphi}
\AtBeginDocument{\mathcode`φ="8000 }

The \AtBeginDocument is important because some other packages might assign a different mathcode.

1
  • I presumed \Umathcode were the modern version of \mathcode, which was incorrect, apparently. Does that mean that each character in XeTeX now has two kinds of mathchar attributes, the plain mathchar and a new Unicode mathchar? From you anwser it seems you can't map one to the other in general.
    – siracusa
    Jun 20, 2017 at 22:00
4

In original TeX, declaring a math character with \mathcode, etc. is the equivalent of declaring a 16-bit integer, where the low order 8 bits are a character code. In the high-order 8 bits, 3 bits are devoted to storing a math class number between 0 and 7, and 4 bits are devoted to recording one of 16 font families, indexed from 0 to 15. That leaves one high-order bit in the 16-bit number. If it is set (i.e., the 0x8000 bit), that declares the math character as an active character. The top four bits of a \mathchar thus record the active bit and the 3-bit class code. So it is not quite correct to talk about "class 8", which doesn't exist. It's only an 8 where a class integer usually appears in the fourth hex digit because of that 16th bit being where it is.

After some years of use, the limitation of only 16 font families (each family being a triplet of individual declared fonts) for math typesetting was becoming cumbersome.

Also, in original TeX, there is no distinction between a single byte character code and an index into a font to find its glyph. But in the Unicode world with OpenType fonts, there is an internal mapping ('cmap' table) from Unicode character to glyph slot number in a font. And it is generally not the identity map, whereas in original TeX, it was always (implicitly) the identity map.

In XeTeX, new primitives \Umathchar and similar were introduced. They treat math character information as a 32-bit integer, rather than a 16-bit integer. This 32-bit integer is divided up into three bit fields whose sizes are 8 bits for the font family code from 0 to 255, 3 bits for the TeX class code from 0 to 7, and 21 bits for a "glyph slot" in that font family.

So the new primitives expect three integers (instead of one) on the input, and they are typically specified as three adjacent hex integers, e.g., "2"41"777 (class 2, family 65, slot 1911). The primitive then combines the integers into a 32-bit word, with appropriate bit shifting and argument range validation. Note that unlike in classic TeX, this syntax would not have to be changed were a TeX interpreter in the future to support more than 8 math classes (in Unicode, there are already more than 8 math classes, but they are boiled down to TeX's eight when XeTeX initializes).

Although the 21-bit field is large enough to hold any Unicode code point value, note that (a) a glyph slot is not a Unicode character, and (b) 21-bits can hold values that are not Unicode code points. Indeed, current OpenType fonts are limited to a maximum of 65536 glyph slots. But perhaps future font formats will support the entirety of Unicode.

So you see the problem. There is no extra bit available in a 32-bit world to use to mark the extended math character as an active math character. But, Unicode does not use up, and likely will not require for a long time, all the possible bit values in a 21-bit integer. In particular, the value 0x1FFFFF (all 21 bits set to 1) is not a legal Unicode code point (Unicode values are in the range 0 to 0x10FFFF).

So XeTeX stores the value 0x1FFFFF (2097151) in the 32-bit word's 21-bit glyph slot field to signify that the extended math character is active. This is a hack, and in a perfect world would not be something that would be exposed to the user. But ... if you execute the following XeTeX code

{\mathcode`@="8000 \the\Umathcodenum`@}

you will find the otherwise mysterious decimal number 2097151 entered into your document layout, as opposed to what you might expect, which would be 32768. Peculiarly though, one can't use "1FFFFF as the third integer in a \Umathcode definition. So

{\Umathcode`@="0"0"1FFFFF}

issues a bogus "bad character code" error (bogus because it's not supposed to be a character, and regardless, it's a glyph slot, not a character). So currently in XeTeX, you have to declare any active character using the classic syntax with the "8000 constant.

Anyway, that's why a class value of 8 isn't legal in a \Umathcode or similar XeTeX command. It's technically not legal in the old TeX world, it just happens to work when shifting the "illegal" class code of 8 into its final place in the integer word, where it sets the high-order bit of the 16-bit math char integer.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .