I want to implement something similar to \detokenize as a Lua function. But there is one case where I am prevented from accomplishing my goal.

\directlua{tex.sprint("[" .. token.get_next().csname .. "]")}\par
\directlua{tex.sprint("[" .. token.get_next().csname .. "]")}\UNDEFINED

The first line above outputs [par] and the other outputs []. Replacing token.get_next().csname by token.scan_csname() doesn't help. How do I obtain the name of an undefined control sequence in LuaTeX?

  • good question, I'd have expected .csname or scan_csname to work, but as you say, they don't. Might be better to ask on the luatex list, see what Hans says! – David Carlisle Oct 26 '16 at 10:16
  • Reading the LuaTeX manual I suspect this is 'by design' behaviour, as it's not actually the user input that token.get_next().csname returns, rather the csname linked to this input where it exists. – Joseph Wright Oct 26 '16 at 11:16
  • 1
    raised on luatex list as tug.org/pipermail/luatex/2016-October/006257.html – David Carlisle Oct 26 '16 at 11:40
  • BTW, I suspect a \detokenize equivalent won't work as you can't ell if a token was an active char (due to dropping of the escape char): ~ and \~ yield the same from token.get_next().csname, for example. – Joseph Wright Oct 26 '16 at 12:48
  • @JosephWright: \directlua{tex.sprint(token.get_next().active and "active character" or "control sequence")} – user114332 Oct 26 '16 at 16:36

Note that you don't get an empty string because \UNDEFINED is not defined but because it has never been scanned by TeX (so there is no internal token of that name)

this plain luatex

\directlua{texio.write_nl("[" .. token.scan_csname() .. "]")}#1%
\directlua{texio.write_nl("[" .. token.get_next().csname .. "]")}#1%



produces a log


showing that just having \UNDEFINED being seen by TeX's #1 argument scanner is enough to get the behaviour you expected.

  • It's a bug, then? – user114332 Oct 26 '16 at 16:52
  • @user114332 no by design (not a design I'd have used myself, especially for .scan_csname which could return the string that it scanned, it's more reasonable for get_next() as there is no token defined for \UNDEFINED until TeX has scanned it. see the email list discussion – David Carlisle Oct 26 '16 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.