LuaTeX analogue of \ifcsname for active characters

In pdfTeX or XeTeX the following code runs through active characters and eventually produces an error such as Invalid code (1114112) (but with 256 in pdfTeX).

\catcode*=13 % active
\count0=32
\def\step{%
\lccode*=\count0
\lowercase{\ifx\undefined*\else\message{\string*}\fi}%
\step
}
\step


In LuaTeX it runs out of memory. My guess is that active characters, like control sequences, are added to the hash table. I would like an analogue of \ifcsname for active characters: that does not add the control sequence to the hash table.

• The exact message I get with luatex is ! TeX capacity exceeded, sorry [number of strings=496604]. – ShreevatsaR Apr 12 '17 at 18:54
• I think this is 'by design' based on Section 2.2.3 of the LuaTeX manual ('Simple single-character csnames'), at least in terms of the fact that actives are added to the hash table. – Joseph Wright Apr 12 '17 at 21:10
• I played with this a bit, and I'm not sure about the guess that active characters being added to the hash table explains the running out of strings. For example: try defining \step as \def\step{\advance\count0 by 1 \catcode\count0=13 \step}. This loops through the whole range (reaching 0x10FFFF=1114111) in both XeTeX and LuaTeX. I think it has something specifically to do with the \lowercase formulation (which BTW could also be written as \lowercase{\ifdefined*\message{\string*}\fi} FWIW). – ShreevatsaR Apr 12 '17 at 21:10
• Running with max_strings=1200000 hash_extra=1200000 luatex seems to work :-) (Just one of them isn't enough, so it confirms that the addition to the hash table is indeed affecting things.) – ShreevatsaR Apr 13 '17 at 0:45
• You can in Lua code do tex.getcatcode(126) (which returns 13 because the tilde is an active character). Is that the sort of thing you were looking for? E.g. one can write a TeX macro \isactive such that \isactive\count0 will do one thing or another depending on whether \count0 corresponds to an active character or not. Or it can even take the actual character instead of a number: \isactive * or \isactive ~. This would fit the title question "LuaTeX analogue of \ifcsname for active characters" but I'm not sure about the "hash table" part (I was able to run it to 1114111 though). – ShreevatsaR Apr 13 '17 at 2:48

The manual says

Active characters are internally implemented as a special type of multi-letter control sequences that uses a prefix that is otherwise impossible to obtain.

Poking around the sources suggests that this prefix is U+FFFF = 239,191,191 as UTF-8, so I think you can do:

\catcode∞=\active
\def∞{and beyond}

\directlua{
for i,v in pairs (tex.hashtokens()) do
if
(
string.byte(v,1,1)==239 and
string.byte(v,2,2)==191 and
string.byte(v,3,3)==191
)
then
texio.write_nl('active char [' ..string.sub(v,4) ..']')
end
end
}

\bye


which produces

active char [^L]
active char [^M]
active char [^Z]
active char [ ]
active char [']
active char [_]
active char [~]
active char [∞]


This will I guess only report characters that have ever had an active definition, if you \let the character to undefined your original will show it as undefined, but this would show the hash entry.

• Thanks. If I understand correctly though, active characters > 127 will be stored in UTF-8 so the total string will have more than 4 bytes, no? – Bruno Le Floch Apr 14 '17 at 3:03
• @BrunoLeFloch updated, dropping the len=4 test and with a non-ascii active char test added. – David Carlisle Apr 14 '17 at 9:27
• Thanks! Now I'm wondering if I'm suffering from an XY problem. What I really want eventually is to use this in l3tl-analysis, which splits a token list into tokens and determines some of their properties: in engines other than XeTeX and pdfTeX I avoid issues of active characters by undefining all of them, but in LuaTeX I'm probably better off looping through tokens on the Lua side. – Bruno Le Floch Apr 14 '17 at 13:54