# Giving arbitrary unicode characters, passed as arguments, a math-active definition?

I've been trying to write a LaTeX3 function to make arbitrary unicode characters math active and give them a definition. I'm using XeLaTeX. My solution inexplicably works for some examples but not for others.

It works for the ascii letter q:

\ExplSyntaxOn
\cs_new_protected:Npn \my_set_math_active:Nn #1 {
\char_set_mathcode:nn {#1} {"8000}
\tl_rescan:nn {\char_set_catcode_active:N #1} {\def #1}
}
\my_set_math_active:Nn q {(test)}
\ExplSyntaxOff

q' is used in $q$. % works

It also works for all non-letters I've tried (uncomplicated ones, anyway; not counting \, #, etc.). But it doesn't work for any letter in the mathematical alphanumeric symbols range. For example, with a double-struck 𝕢:

\my_set_math_active:Nn 𝕢 {(test)} % error

I get the error ! Missing control sequence inserted.. It doesn't recognize that the letter I'm trying to define is an active character. The following, however, works:

\char_set_mathcode:nn {𝕢} {"8000}
{\char_set_catcode_active:N 𝕢 \gdef 𝕢 {(test)}}
\ExplSyntaxOff

𝕢' is used in $𝕢$. % works, sort of

But this solution has its own problems:

1. If 𝕢 appears in its own definition, it will have catcode 13 and thus cause non-termination.

2. It makes the definition global. I'd like to avoid global definitions whenever possible.

So why doesn't the original code work? I can only guess that 𝕢 doesn't survive \tl_rescan:nn. But I'm not sure what to try at this point.

I'd appreciate some help!

## Motivation

I want to let you know why I need this: I'm basically writing a 'math-mode lexical analyzer'. That is to say, I want to make LaTeX a bit smarter and more flexible in interpreting character sequences in math mode. This will allow, among other things:

1. combinations of symbols to be interpreted as a single command. This is especially useful with unicode. I am, for example, defining the combination ──► as \longrightarrow, so that it is visually distinguishable from (\rightarrow) in a monospaced editor font. The lexer uses maximal munch matching, so I can also define ──►► as \longtwoheadrightarrow.

2. (strings of) letters to be understood in the semantic context of your theory, automatically give them the right font, and perhaps hyperlink them to their definition, all with minimal input. For example, in $sin(x)$, LaTeX might automatically set sin upright and x in italics.

This already works. But it's a shame that the full unicode range cannot yet be used.

I am still particularly interested why the code above works for q but not for 𝕢. It would allow the definitions to remain local. Below is a somewhat more elegant way to code it, but still with the same result (without the code for \normal):

\tl_new:N \l__my_active_tl
\cs_new_protected:Nn \my_set_math_active:Nn {
\tl_set_rescan:Nnn
\l__my_active_tl
{\char_set_catcode_active:N #1}
{#1}
\exp_last_unbraced:NV
\cs_set:Npn~\l__my_active_tl~{ #2 }
\char_set_mathcode:nn {#1} {"8000}
}

\my_set_math_active:Nn q {(testa)} % fine
\my_set_math_active:Nn 𝕢 {(testb)} % error

Is it a \tl_set_rescan:nn bug?

-

You have found a bug in XeTeX's implementation of \scantokens (the underlying primitive used for LaTeX3's \tl_rescan:nn) for characters beyond the BMP.

Running the following through (plain) LuaTeX yields (./test.tex ****120162,32**** ), the rightful character code of 𝕢 followed by that of a space (which follows #2 in the definition of \test).

Running it throught (plain) XeTeX yields (./test.tex ****55349,56674**** ), which are the two pairs of bytes appearing in the UTF-16 representation of 𝕢 (at least they're in the right ballpark). Basically, rescanning transforms 𝕢 into a pair of characters. Somehow, though, 𝕢 can safely go through being written to a file and input back: the problem really seems specific to \scantokens.

\def\test#1#2.{\message{****\number#1,\number#2 ****}}
\scantokens{\test 𝕢.}
\bye

-
Cool! I was about to reach the same conclusion since I couldn't make this work with \scantokens either. Though I could not have written such a clear test. --- Next to reporting the bug, I suppose I could also manually correct for it. –  mhelvens Aug 5 '13 at 7:13

There is already a function for globally assigning a meaning to an active character, without resorting to \tl_rescan:nn.

\documentclass{article}
\usepackage{xparse}
\usepackage{fontspec}
\setmainfont[Ligatures=TeX]{STIXGeneral}

\ExplSyntaxOn
\cs_new_protected:Npn \my_set_math_active:Nn #1 #2
{
\AtBeginDocument{
\char_set_mathcode:nn {#1} { "8000 }
}
\group_begin:
\char_gset_active:Npn #1 { #2 }
\group_end:
}
\my_set_math_active:Nn q {(testa)}
\my_set_math_active:Nn 𝕢 {(testb)}
\ExplSyntaxOff

\begin{document}

q' is used in $q$.

𝕢' is used in $𝕢$.

\end{document}

You can't use directly a math active character in its definition, because an infinite loop will result. It has nothing to do with active characters; with the classic

{\catcode?=\active \xdef?{(\string?)}}
\mathcode?="8000

the input $?$ would explode even if ? is not active, because it's math active.

There are workarounds. Here's a way: if you want to use a character in its replacement text when made math active, use \normal:

\documentclass{article}
\usepackage{xparse}
\usepackage{unicode-math}
\setmainfont[Ligatures=TeX]{STIXGeneral}
\setmathfont{XITS Math}

\ExplSyntaxOn
\cs_new_protected:Npn \helvens_set_math_active:Nn #1 #2
{
\group_begin:
\char_gset_active:Npn #1 { #2 }
\group_end:
\cs_set:cpx { helvens_old_#1 }
{ \Umathcharnum \int_eval:n { \Umathcodenum#1 } ~ } % a space for terminating the number
\char_set_mathcode:nn {#1} { "8000 }
}
\NewDocumentCommand{\setmathactive}{mm}
{
\helvens_set_math_active:Nn #1 { #2 }
}
\NewDocumentCommand{\normal}{m}
{
\use:c { helvens_old_#1 }
}
\ExplSyntaxOff

\setmathactive{q}{(\normal{q})}
\setmathactive{𝕢}{(\normal{𝕢})}

\begin{document}

q' is used in $q$.

𝕢' is used in $𝕢$.

And $\normal{𝕢}$ works in math.

\end{document}

-
Ah, I didn't realize that function existed. However, this approach appears to be equivalent to my last code-example and therefore has the same problems: (1) if 𝕢 appears in its own definition, it will have catcode 13 (is this a feature or a bug?) and thus cause non-termination and (2) it makes the definition global. I'd like to avoid global definitions whenever possible. --- Also two (unrelated) questions if I may: (a) Why load xparse rather than expl3? (b) Why use \AtBeginDocument here? Thanks! –  mhelvens Aug 4 '13 at 17:23
@mhelvens Of course you can't use the character in the body of the replacement text, because it has been activated. Also using \token_to_str:N won't be good, because in math mode this would cause infinite recursion. An example of intended usage is necessary. The \AtBeginDocument is "traditional" (amsmath related). About xparse: it's probably better to define a user level command, rather than making the definitions with programmer's level functions. –  egreg Aug 4 '13 at 18:02
Indeed, that's the issue. Inside the replacement text it has been activated (catcode 13), but it is supposed to only have been math activated. A test case: \my_set_math_active:Nn 𝕢 {(\hbox{𝕢})} should not cause problems. --- For practical use I intend to provide a simple function to return the 'non-math-activated' version of a character (e.g. \old{𝕢}) that authors can use in the replacement text. Will it not be easier to define \old in terms of math codes than in terms of \tl_rescan? –  mhelvens Aug 4 '13 at 18:13
Hi @egreg: Thanks for the update! It seems to work well, but I'll test it more thoroughly in the coming days. --- I still wonder how we might keep the definitions local, however. I have added another code sample to my question that works for q but fails for 𝕢. I'd love to know why this is. –  mhelvens Aug 4 '13 at 22:44
Thanks for your effort in finding an acceptable solution to the problem – the one that I am now using. I'm giving you the bounty. However, Bruno's answer is actually the 'correct' one. I'll move back to my old implementation when the bug in XeTeX is fixed, in order to maintain local scoping. So I'm giving his answer the green tick. Cheers! –  mhelvens Aug 11 '13 at 12:27