Using \directlua and tex.print, it is possible to send strings of characters back and forth between TeX and Lua. Let's take an example: starting with a macro \A, I pass its replacement text to Lua and back, defining \B

\edef\A{ab\string c}
  \message{Identical :)!}
  \message{Different :(.}

The two macros appear identical, but they aren't: the weird/special combination of category codes (letter and other) is lost before reaching Lua, and tokenization on the return trip is done using the category code régime in effect when \directlua is performed.

Can the \directlua line be replaced by some other Lua code which would be able to define \B to be identical to \A, including category codes? Of course,


would work (by delaying the expansion of \A until after the passage through LuaTeX), but my goal would be to perform some complicated operations on the token lists on the Lua side.

  • 4
    When you say 'category codes have been lost', I think it would be more accurate to say 'tokenization takes place using the catcode régime applicable when \directlua is executed'. (The tokens still have category codes, just not the ones you expected!)
    – Joseph Wright
    Feb 21, 2012 at 17:36
  • @Joseph Thanks, I clarified that (and changed \A a little, hopefully that makes things clearer). Btw, the French word is régime (we also had that typo in the LaTeX3 doc at some point). Feb 21, 2012 at 17:43
  • Drat - edited :-)
    – Joseph Wright
    Feb 21, 2012 at 17:46
  • The answer(s) below only handle passing tokens from Lua to TeX (tprint), not passing tokens from TeX to Lua. I wrote an answer in luatex - How can I get the value of a token list (tl) variable in Lua? - TeX - LaTeX Stack Exchange, but there are some other uses cases (perhaps something can be built with runtoks() / coroutines to run TeX from Lua / scan_* functions. There's also put_token.
    – user202729
    Jan 6, 2022 at 6:44
  • Actually I think scan_toks can do as well, there's roughly one answer on the site that use it so far and documentation isn't great, tex.stackexchange.com/a/555222/250119, pair with expandafter and some other and can read value of token list as well. Print table of token objects from Lua to TeX is easy
    – user202729
    Jan 8, 2022 at 15:33

3 Answers 3


You can play with tex.print() and catcode tables. The result of


gives "Identical :)". This is, because -2 is a predefined catcode table where all characters have the catcode "other" (12) except for space (10) assigned. So you get the same as your edef in TeX.

  • What is the difference between tex.print and tex.tprint? And how general is that? It seems that I would need one catcode table for each category code; not much simpler than the TeX way of building arbitrary token lists. Feb 21, 2012 at 20:05
  • 3
    @BrunoLeFloch the difference between the two functions is that tex.tprint() can output many strings with different catcode tables with one function call, while with tex.print() you need one for each different cc table. You can define catcode tables in advance with the catcodes you want to have, there are a few predefined ones. -- I guess it all depends on what you want to do. I don't understand your question about how general this is.
    – topskip
    Feb 21, 2012 at 20:11
  • well, in my example I only have characters with either the normal catcode régime (a and b are letters, if I had put #, it would be a macro parameter) and the "-2" table (where all characters have catcode other, here only used for c). The general question would be how to build a token list with arbitrary combinations of character and category code. I do that in l3regex, and doing it in TeX is slightly tricky. Feb 21, 2012 at 20:18
  • Actually, your answer only addresses half of the problem: going from Lua to TeX and getting tokens with various catcodes. The other direction seems tougher: if I have a list of tokens in TeX, how can I transmit information about all the catcodes to Lua? I could analyse the token list on the TeX side, and transmit info about each token, but that ends up mixing TeX and Lua in ugly ways I believe. Feb 21, 2012 at 20:22
  • If the characters of both (normal catcode régime and the "other" catcodes) don't overlap, you can just define your own catcodes and store them in a catcodetable (with a number n). Then you can use tex.print(n, "string") and it will use your predefined cctable.
    – topskip
    Feb 21, 2012 at 20:23

Using the token_filter callback and the token library (as Patrick kindly mentionned to me in comments to his answer) allows to build arbitrary token lists and insert them in the input stream. The callback should only be called once, it disables itself with callback.register(..., nil).

              callback.register ('token_filter', nil)
                  token.create(\number`\a, 11),
                  token.create(\number`\b, 11),
                  token.create(\number`\c, 12)

This answer only covers the Lua to TeX part. It should be possible to use token.expand() to do the TeX to Lua step, but I don't know how.


A quick summary/"cheat sheet" of the relevant commands (answer the question in title).

  • TeX (convenience function): with \usepackage{luacode} it's possible to use \begin{luacode*} ... \end{luacode*} to write Lua code without worrying about catcode.

  • Lua built-in has inspect() function used to print out some basic information about any (userdata) object / pretty-print tables etc..

  • Tokens in Lua:

    • To get token from Lua: (see documentation for more details)

      • token.scan_toks(): get an argument (\toks0={...}-style, must be wrapped in braces, drop outermost brace, can contain control sequences not in the hash table).

      • token.scan_toks(false, true): get an argument (\expanded{...}-style, must be wrapped in braces)

      • token.create("<control sequence name>"), token.create(int charcode, int catcode)


        • create the impossible control sequence if it's not in the hash table
        • the null control sequence (control sequence name = "") cannot be created this way
      • token.get_next() returns the next token in the input stream, must be in the hash table (*1)

      • token.new: (presumably) create a "new" (frozen) token with a definite meaning. There's an example usage at https://tex.stackexchange.com/a/498118/250119

    • Token format:

      • check .csname
        • if is nil: explicit character (except some special cases)

          • .mode = charcode (might be .index instead in newer versions)

          • .cmdname ∈ (left_brace right_brace math_shift tab_mark mac_param sup_mark sub_mark spacer letter other_char).

            • if it's not in this list, it could be some internal tokens such as the \endlocalcontrol inserted inside a tex.runtoks(), which has .mode = 10 and .cmdname = extension, but .csname nil
        • else:

          • if .active: active character (charcode is in .csname, not .mode), else: control sequence

          • if .csname is the exact string "\csname\endcsname" (unusually include the backslashes, and regardless of actual value of escapechar) and .tok = 0x20000000: is null control sequence token

          • if (...) there are many internal tokens, if you want to be very careful, as the LuaTeX manual stated, create a token and get its .tok value.

          • else: is the active character/control sequence with the given csname.

          • check .cmdname for its meaning, can be one of the following

            token.commands() = {

            (call long_call outer_call long_outer_call) are user-defined macro.

            In this case .mode (or .index) will have some cmdname-specific meaning.

    • To send back tokens from Lua to TeX:

      • tex.print(...): (with "trailing newline" i.e. on a separate line)

        • (token1)

        • { token1, token2 }

        • (string content)

        • (int catcodetablenumber, string content)

          (catcodetablenumber special values: -1 is current, -2 is detokenized)

        • ([int catcodetablenumber], {string1, string2, ...})

      • tex.sprint: same as above but without "trailing newline"

      • tex.tprint: shortcut

            {int catcodetablenumber1, string content1},
            {int catcodetablenumber2, string content2},
      • tex.cprint(int catcode, string content)

      • token.put_next( token1, token2 ), token.put_next{ token1, token2 }

        Note: you usually don't mix token.put_next and tex.print (print always come before all put_next!)

  • Execute TeX code in Lua:

  • Execute Lua code in TeX:

    • \directlua{lua code to be executed... (fully expanded similar to \message or \expanded, # are doubled)}
    • Pass argument in with \luaescapestring{content... (fully expanded similar to \message or \expanded, # are not doubled)}
    • Or leave the argument in the input stream and let Lua pick them up with e.g. token.scan_toks or token.get_next etc.

(*1): otherwise it becomes \IMPOSSIBLE. A simple \futurelet can be used here safely, but there's certain overhead. Use scan_toks() if possible.

E.g. Lua function to increment the char code of each token by 1, works across all group levels and takes one expansion step. (demo for "take token list and do complex manipulation")

function f()
    local tokens=token.scan_toks()
    for i=1,#tokens do
        if tokens[i].cmdname=="letter" then
            tokens[i]=token.create(tokens[i].mode+1, 11)
    tokens[#tokens+1]=token.create(string.byte("}"), 2)
        token.create(string.byte("{"), 1),

\directlua{f()}{ab{c}123}  % eventually expand to bc{d}123

Another examples: https://tex.stackexchange.com/a/555222/250119


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .