# Can the Lua part of LuaTeX know about tokens?

Using \directlua and tex.print, it is possible to send strings of characters back and forth between TeX and Lua. Let's take an example: starting with a macro \A, I pass its replacement text to Lua and back, defining \B

\edef\A{ab\string c}
\directlua{tex.print("\noexpand\\def\noexpand\\B{\unexpanded\expandafter{\A}}")}
\show\A
\show\B
\ifx\A\B
\message{Identical :)!}
\else
\message{Different :(.}
\fi
\bye


The two macros appear identical, but they aren't: the weird/special combination of category codes (letter and other) is lost before reaching Lua, and tokenization on the return trip is done using the category code régime in effect when \directlua is performed.

Can the \directlua line be replaced by some other Lua code which would be able to define \B to be identical to \A, including category codes? Of course,

\directlua{\unexpanded{tex.print("\\edef\\B{\\unexpanded\\expandafter{\\A}}")}}


would work (by delaying the expansion of \A until after the passage through LuaTeX), but my goal would be to perform some complicated operations on the token lists on the Lua side.

• When you say 'category codes have been lost', I think it would be more accurate to say 'tokenization takes place using the catcode régime applicable when \directlua is executed'. (The tokens still have category codes, just not the ones you expected!) – Joseph Wright Feb 21 '12 at 17:36
• @Joseph Thanks, I clarified that (and changed \A a little, hopefully that makes things clearer). Btw, the French word is régime (we also had that typo in the LaTeX3 doc at some point). – Bruno Le Floch Feb 21 '12 at 17:43
• Drat - edited :-) – Joseph Wright Feb 21 '12 at 17:46

You can play with tex.print() and catcode tables. The result of

tex.tprint({"\\def\\B{ab"},{-2,"c"},{"}"})


gives "Identical :)". This is, because -2 is a predefined catcode table where all characters have the catcode "other" (12) except for space (10) assigned. So you get the same as your edef in TeX.

• What is the difference between tex.print and tex.tprint? And how general is that? It seems that I would need one catcode table for each category code; not much simpler than the TeX way of building arbitrary token lists. – Bruno Le Floch Feb 21 '12 at 20:05
• @BrunoLeFloch the difference between the two functions is that tex.tprint() can output many strings with different catcode tables with one function call, while with tex.print() you need one for each different cc table. You can define catcode tables in advance with the catcodes you want to have, there are a few predefined ones. -- I guess it all depends on what you want to do. I don't understand your question about how general this is. – topskip Feb 21 '12 at 20:11
• well, in my example I only have characters with either the normal catcode régime (a and b are letters, if I had put #, it would be a macro parameter) and the "-2" table (where all characters have catcode other, here only used for c). The general question would be how to build a token list with arbitrary combinations of character and category code. I do that in l3regex, and doing it in TeX is slightly tricky. – Bruno Le Floch Feb 21 '12 at 20:18
• Actually, your answer only addresses half of the problem: going from Lua to TeX and getting tokens with various catcodes. The other direction seems tougher: if I have a list of tokens in TeX, how can I transmit information about all the catcodes to Lua? I could analyse the token list on the TeX side, and transmit info about each token, but that ends up mixing TeX and Lua in ugly ways I believe. – Bruno Le Floch Feb 21 '12 at 20:22
• If the characters of both (normal catcode régime and the "other" catcodes) don't overlap, you can just define your own catcodes and store them in a catcodetable (with a number n). Then you can use tex.print(n, "string") and it will use your predefined cctable. – topskip Feb 21 '12 at 20:23

Using the token_filter callback and the token library (as Patrick kindly mentionned to me in comments to his answer) allows to build arbitrary token lists and insert them in the input stream. The callback should only be called once, it disables itself with callback.register(..., nil).

\expandafter\def\expandafter\B\expandafter{%
\directlua
{
callback.register
(
'token_filter',
function()
callback.register ('token_filter', nil)
return
{
token.create(\number\a, 11),
token.create(\number\b, 11),
token.create(\number\c, 12)
}
end
)
}%
}
\show\B
\def\test#1#2#3{\show#1\show#2\show#3}
\expandafter\test\B


This answer only covers the Lua to TeX part. It should be possible to use token.expand() to do the TeX to Lua step, but I don't know how.

• Apparently, token.expand()` is broken. I will try to update this answer when it is fixed. – Bruno Le Floch Aug 3 '12 at 15:06