Giving arbitrary unicode characters, passed as arguments, a math-active definition?

Question

I've been trying to write a LaTeX3 function to make arbitrary unicode characters math active and give them a definition. I'm using XeLaTeX. My solution inexplicably works for some examples but not for others.

It works for the ascii letter q:

\ExplSyntaxOn
\cs_new_protected:Npn \my_set_math_active:Nn #1 {
    \char_set_mathcode:nn {`#1} {"8000}
    \tl_rescan:nn {\char_set_catcode_active:N #1} {\def #1}
}
\my_set_math_active:Nn q {(test)}
\ExplSyntaxOff

`q' is used in $q$. % works

It also works for all non-letters I've tried (uncomplicated ones, anyway; not counting \, #, etc.). But it doesn't work for any letter in the mathematical alphanumeric symbols range. For example, with a double-struck 𝕢:

\my_set_math_active:Nn 𝕢 {(test)} % error

I get the error ! Missing control sequence inserted.. It doesn't recognize that the letter I'm trying to define is an active character. The following, however, works:

\char_set_mathcode:nn {`𝕢} {"8000}
{\char_set_catcode_active:N 𝕢 \gdef 𝕢 {(test)}}
\ExplSyntaxOff

`𝕢' is used in $𝕢$. % works, sort of

But this solution has its own problems:

1. If 𝕢 appears in its own definition, it will have catcode 13 and thus cause non-termination.

2. It makes the definition global. I'd like to avoid global definitions whenever possible.

So why doesn't the original code work? I can only guess that 𝕢 doesn't survive \tl_rescan:nn. But I'm not sure what to try at this point.

I'd appreciate some help!

---

Motivation

I want to let you know why I need this: I'm basically writing a 'math-mode lexical analyzer'. That is to say, I want to make LaTeX a bit smarter and more flexible in interpreting character sequences in math mode. This will allow, among other things:

1. combinations of symbols to be interpreted as a single command. This is especially useful with unicode. I am, for example, defining the combination ──► as \longrightarrow, so that it is visually distinguishable from → (\rightarrow) in a monospaced editor font. The lexer uses maximal munch matching, so I can also define ──►► as \longtwoheadrightarrow.

2. (strings of) letters to be understood in the semantic context of your theory, automatically give them the right font, and perhaps hyperlink them to their definition, all with minimal input. For example, in $sin(x)$, LaTeX might automatically set sin upright and x in italics.

This already works. But it's a shame that the full unicode range cannot yet be used.

---

I am still particularly interested why the code above works for q but not for 𝕢. It would allow the definitions to remain local. Below is a somewhat more elegant way to code it, but still with the same result (without the code for \normal):

\tl_new:N \l__my_active_tl
\cs_new_protected:Nn \my_set_math_active:Nn {
    \tl_set_rescan:Nnn
        \l__my_active_tl
        {\char_set_catcode_active:N #1}
        {#1}
    \exp_last_unbraced:NV
        \cs_set:Npn~\l__my_active_tl~{ #2 }
    \char_set_mathcode:nn {`#1} {"8000}
}

\my_set_math_active:Nn q {(testa)} % fine
\my_set_math_active:Nn 𝕢 {(testb)} % error

Is it a \tl_set_rescan:nn bug?

Answer 1

EDIT: the bug has been fixed; I haven't checked when.

You have found a bug in XeTeX's implementation of \scantokens (the underlying primitive used for LaTeX3's \tl_rescan:nn) for characters beyond the BMP.

Running the following through (plain) LuaTeX yields (./test.tex ****120162,32**** ), the rightful character code of 𝕢 followed by that of a space (which follows #2 in the definition of \test).

Running it throught (plain) XeTeX yields (./test.tex ****55349,56674**** ), which are the two pairs of bytes appearing in the UTF-16 representation of 𝕢 (at least they're in the right ballpark). Basically, rescanning transforms 𝕢 into a pair of characters. Somehow, though, 𝕢 can safely go through being written to a file and input back: the problem really seems specific to \scantokens.

\def\test#1#2.{\message{****\number`#1,\number`#2 ****}}
\scantokens{\test 𝕢.}
\bye

Please report.

Answer 2

There is already a function for globally assigning a meaning to an active character, without resorting to \tl_rescan:nn.

\documentclass{article}
\usepackage{xparse}
\usepackage{fontspec}
\setmainfont[Ligatures=TeX]{STIXGeneral}

\ExplSyntaxOn
\cs_new_protected:Npn \my_set_math_active:Nn #1 #2
 {
  \AtBeginDocument{
    \char_set_mathcode:nn {`#1} { "8000 }
  }
  \group_begin:
  \cs_set_protected:Nn \my_set_math_active_temp: { #2 }
  \char_gset_active_eq:NN #1 \my_set_math_active_temp:
  \group_end:
}
\my_set_math_active:Nn q {(testa)}
\my_set_math_active:Nn 𝕢 {(testb)}
\ExplSyntaxOff

\begin{document}

`q' is used in $q$.

`𝕢' is used in $𝕢$.


\end{document}

---

You can't use directly a math active character in its definition, because an infinite loop will result. It has nothing to do with active characters; with the classic

{\catcode`?=\active \xdef?{(\string?)}}
\mathcode`?="8000

the input $?$ would explode even if ? is not active, because it's math active.

There are workarounds. Here's a way: if you want to use a character in its replacement text when made math active, use \normal:

\documentclass{article}
\usepackage{xparse}
\usepackage{unicode-math}
\setmainfont[Ligatures=TeX]{STIXGeneral}
\setmathfont{XITS Math}

\ExplSyntaxOn
\cs_new_protected:Npn \helvens_set_math_active:Nn #1 #2
 {
  \group_begin:
  \cs_set_protected:Nn \helvens_set_math_active_temp: { #2 }
  \char_gset_active_eq:NN #1 \helvens_set_math_active_temp:
  \group_end:
  \cs_set:cpx { helvens_old_#1 }
   { \Umathcharnum \int_eval:n { \Umathcodenum`#1 } ~ } % a space for terminating the number
  \char_set_mathcode:nn {`#1} { "8000 }
 }
\NewDocumentCommand{\setmathactive}{mm}
 {
  \helvens_set_math_active:Nn #1 { #2 }
 }
\NewDocumentCommand{\normal}{m}
 {
  \use:c { helvens_old_#1 }
 }
\ExplSyntaxOff

\setmathactive{q}{(\normal{q})}
\setmathactive{𝕢}{(\normal{𝕢})}

\begin{document}

`q' is used in $q$.

`𝕢' is used in $𝕢$.

And $\normal{𝕢}$ works in math.

\end{document}