# Can the Lua part of LuaTeX know about tokens?

Using \directlua and tex.print, it is possible to send strings of characters back and forth between TeX and Lua. Let's take an example: starting with a macro \A, I pass its replacement text to Lua and back, defining \B

\edef\A{ab\string c}
\directlua{tex.print("\noexpand\\def\noexpand\\B{\unexpanded\expandafter{\A}}")}
\show\A
\show\B
\ifx\A\B
\message{Identical :)!}
\else
\message{Different :(.}
\fi
\bye


The two macros appear identical, but they aren't: the weird/special combination of category codes (letter and other) is lost before reaching Lua, and tokenization on the return trip is done using the category code régime in effect when \directlua is performed.

Can the \directlua line be replaced by some other Lua code which would be able to define \B to be identical to \A, including category codes? Of course,

\directlua{\unexpanded{tex.print("\\edef\\B{\\unexpanded\\expandafter{\\A}}")}}


would work (by delaying the expansion of \A until after the passage through LuaTeX), but my goal would be to perform some complicated operations on the token lists on the Lua side.

• When you say 'category codes have been lost', I think it would be more accurate to say 'tokenization takes place using the catcode régime applicable when \directlua is executed'. (The tokens still have category codes, just not the ones you expected!) Feb 21, 2012 at 17:36
• @Joseph Thanks, I clarified that (and changed \A a little, hopefully that makes things clearer). Btw, the French word is régime (we also had that typo in the LaTeX3 doc at some point). Feb 21, 2012 at 17:43
• Drat - edited :-) Feb 21, 2012 at 17:46
• The answer(s) below only handle passing tokens from Lua to TeX (tprint), not passing tokens from TeX to Lua. I wrote an answer in luatex - How can I get the value of a token list (tl) variable in Lua? - TeX - LaTeX Stack Exchange, but there are some other uses cases (perhaps something can be built with runtoks() / coroutines to run TeX from Lua / scan_* functions. There's also put_token. Jan 6 at 6:44
• Actually I think scan_toks can do as well, there's roughly one answer on the site that use it so far and documentation isn't great, tex.stackexchange.com/a/555222/250119, pair with expandafter and some other and can read value of token list as well. Print table of token objects from Lua to TeX is easy Jan 8 at 15:33

You can play with tex.print() and catcode tables. The result of

tex.tprint({"\\def\\B{ab"},{-2,"c"},{"}"})


gives "Identical :)". This is, because -2 is a predefined catcode table where all characters have the catcode "other" (12) except for space (10) assigned. So you get the same as your edef in TeX.

• What is the difference between tex.print and tex.tprint? And how general is that? It seems that I would need one catcode table for each category code; not much simpler than the TeX way of building arbitrary token lists. Feb 21, 2012 at 20:05
• @BrunoLeFloch the difference between the two functions is that tex.tprint() can output many strings with different catcode tables with one function call, while with tex.print() you need one for each different cc table. You can define catcode tables in advance with the catcodes you want to have, there are a few predefined ones. -- I guess it all depends on what you want to do. I don't understand your question about how general this is. Feb 21, 2012 at 20:11
• well, in my example I only have characters with either the normal catcode régime (a and b are letters, if I had put #, it would be a macro parameter) and the "-2" table (where all characters have catcode other, here only used for c). The general question would be how to build a token list with arbitrary combinations of character and category code. I do that in l3regex, and doing it in TeX is slightly tricky. Feb 21, 2012 at 20:18
• Actually, your answer only addresses half of the problem: going from Lua to TeX and getting tokens with various catcodes. The other direction seems tougher: if I have a list of tokens in TeX, how can I transmit information about all the catcodes to Lua? I could analyse the token list on the TeX side, and transmit info about each token, but that ends up mixing TeX and Lua in ugly ways I believe. Feb 21, 2012 at 20:22
• If the characters of both (normal catcode régime and the "other" catcodes) don't overlap, you can just define your own catcodes and store them in a catcodetable (with a number n). Then you can use tex.print(n, "string") and it will use your predefined cctable. Feb 21, 2012 at 20:23

Using the token_filter callback and the token library (as Patrick kindly mentionned to me in comments to his answer) allows to build arbitrary token lists and insert them in the input stream. The callback should only be called once, it disables itself with callback.register(..., nil).

\expandafter\def\expandafter\B\expandafter{%
\directlua
{
callback.register
(
'token_filter',
function()
callback.register ('token_filter', nil)
return
{
token.create(\number\a, 11),
token.create(\number\b, 11),
token.create(\number\c, 12)
}
end
)
}%
}
\show\B
\def\test#1#2#3{\show#1\show#2\show#3}
\expandafter\test\B


This answer only covers the Lua to TeX part. It should be possible to use token.expand() to do the TeX to Lua step, but I don't know how.

• Apparently, token.expand() is broken. I will try to update this answer when it is fixed. Aug 3, 2012 at 15:06
• Also looks like that token_filter is removed long ago. Mar 28 at 20:58

A quick summary/"cheat sheet" of the relevant commands (answer the question in title).

• TeX (convenience function): with \usepackage{luacode} it's possible to use \begin{luacode*} ... \end{luacode*} to write Lua code without worrying about catcode.

• Lua built-in has inspect() function used to print out some basic information about any (userdata) object / pretty-print tables etc..

• Tokens in Lua:

• To get token from Lua: (see documentation for more details)

• token.scan_toks(): get an argument (#1-style, must be wrapped in braces, drop outermost brace, can contain never-seen control sequences).

• token.create("<control sequence name>"), token.create(int charcode, int catcode)

Note create the impossible control sequence if it's never-seen

• token.get_next() returns the next token in the input stream, must not be never-seen control sequence (*1)

• token.new: ???

• Token format:

• check .csname
• if is nil: explicit character

• .mode = charcode
• .cmdname ∈ (left_brace right_brace math_shift tab_mark mac_param sup_mark sub_mark spacer letter other_char).
• else:

• if .active: active character (charcode is in .csname, not .mode), else: control sequence

• if .csname is the exact string "\csname\endcsname" and .tok = 0x20000000: is null control sequence token

• else: is the active character/control sequence with the given csname.

• check .cmdname for its meaning, can be one of the following

token.commands() = {
[0]="relax",
"left_brace",
"right_brace",
"math_shift",
"tab_mark",
"car_ret",
"mac_param",
"sup_mark",
"sub_mark",
"endv",
"spacer",
"letter",
"other_char",
"par_end",
"stop",
"delim_num",
"char_num",
"math_char_num",
"mark",
"node",
"xray",
"make_box",
"hmove",
"vmove",
"un_hbox",
"un_vbox",
"remove_item",
"hskip",
"vskip",
"mskip",
"kern",
"mkern",
"halign",
"valign",
"no_align",
"vrule",
"hrule",
"novrule",
"nohrule",
"insert",
"ignore_spaces",
"after_assignment",
"after_group",
"break_penalty",
"start_par",
"ital_corr",
"accent",
"math_accent",
"discretionary",
"eq_no",
"left_right",
"math_comp",
"limit_switch",
"above",
"math_style",
"math_choice",
"non_script",
"vcenter",
"case_shift",
"message",
"normal",
"extension",
"option",
"lua_function_call",
"lua_bytecode_call",
"lua_call",
"in_stream",
"begin_group",
"end_group",
"omit",
"ex_space",
"boundary",
"super_sub_script",
"no_super_sub_script",
"math_shift_cs",
"end_cs_name",
"char_ghost",
"assign_local_box",
"char_given",
"math_given",
"xmath_given",
"last_item",
"toks_register",
"assign_toks",
"assign_int",
"assign_attr",
"assign_dimen",
"assign_glue",
"assign_mu_glue",
"assign_font_dimen",
"assign_font_int",
"assign_hang_indent",
"set_aux",
"set_prev_graf",
"set_page_dimen",
"set_page_int",
"set_box_dimen",
"set_tex_shape",
"set_etex_shape",
"def_char_code",
"def_del_code",
"extdef_math_code",
"extdef_del_code",
"def_family",
"set_math_param",
"set_font",
"def_font",
"register",
"assign_box_direction",
"assign_box_dir",
"assign_direction",
"assign_dir",
"combinetoks",
"multiply",
"divide",
"prefix",
"let",
"shorthand_def",
"def_lua_call",
"def",
"set_box",
"hyph_data",
"set_interaction",
"letterspace_font",
"expand_font",
"copy_font",
"set_font_id",
"undefined_cs",
"expand_after",
"no_expand",
"input",
"lua_expandable_call",
"lua_local_call",
"if_test",
"fi_or_else",
"cs_name",
"convert",
"variable",
"feedback",
"the",
"top_bot_mark",
"call",
"long_call",
"outer_call",
"long_outer_call",
"end_template",
"dont_expand",
"glue_ref",
"shape_ref",
"box_ref",
"data",
}


(call long_call outer_call long_outer_call) are user-defined macro.

• To send back tokens from Lua to TeX:

• tex.print(...): (with "trailing newline" i.e. on a separate line)

• (token1)

• { token1, token2 }

• (string content)

• (int catcodetablenumber, string content)

(catcodetablenumber special values: -1 is current, -2 is detokenized)

• ([int catcodetablenumber], {string1, string2, ...})

• tex.sprint: same as above but without "trailing newline"

• tex.tprint: shortcut

tex.tprint(
{int catcodetablenumber1, string content1},
{int catcodetablenumber2, string content2},
...
)

• tex.cprint(int catcode, string content)

• token.put_next( token1, token2 ), token.put_next{ token1, token2 }

Note: you usually don't mix token.put_next and tex.print (print always come before all put_next!)

• Execute TeX code in Lua:

• Execute Lua code in TeX:

• \directlua{lua code to be executed... (x-expanded)}
• Pass argument in with \luaescapestring{content... (x-expanded)}
• Or leave the argument in the input stream and let Lua pick them up with e.g. token.scan_toks or token.get_next etc.

(*1): otherwise it becomes \IMPOSSIBLE. A simple \futurelet can be used here safely, but there's certain overhead. Use scan_toks() if possible.

E.g. Lua function to increment the char code of each token by 1, works across all group levels and takes one expansion step. (demo for "take token list and do complex manipulation")

function f()
local tokens=token.scan_toks()
for i=1,#tokens do
if tokens[i].cmdname=="letter" then
tokens[i]=token.create(tokens[i].mode+1, 11)
end
end
tokens[#tokens+1]=token.create(string.byte("}"), 2)
token.put_next{
token.create("pretty:n"),
token.create(string.byte("{"), 1),
table.unpack(tokens)
}
end

\directlua{f()}{ab{c}123}  % eventually expand to bc{d}123

`

Another examples: https://tex.stackexchange.com/a/555222/250119