Can the Lua part of LuaTeX know about tokens?

Question

Using \directlua and tex.print, it is possible to send strings of characters back and forth between TeX and Lua. Let's take an example: starting with a macro \A, I pass its replacement text to Lua and back, defining \B

\edef\A{ab\string c}
\directlua{tex.print("\noexpand\\def\noexpand\\B{\unexpanded\expandafter{\A}}")}
\show\A
\show\B
\ifx\A\B
  \message{Identical :)!}
\else
  \message{Different :(.}
\fi
\bye

The two macros appear identical, but they aren't: the weird/special combination of category codes (letter and other) is lost before reaching Lua, and tokenization on the return trip is done using the category code régime in effect when \directlua is performed.

Can the \directlua line be replaced by some other Lua code which would be able to define \B to be identical to \A, including category codes? Of course,

\directlua{\unexpanded{tex.print("\\edef\\B{\\unexpanded\\expandafter{\\A}}")}}

would work (by delaying the expansion of \A until after the passage through LuaTeX), but my goal would be to perform some complicated operations on the token lists on the Lua side.

Answer 1

You can play with tex.print() and catcode tables. The result of

tex.tprint({"\\def\\B{ab"},{-2,"c"},{"}"})

gives "Identical :)". This is, because -2 is a predefined catcode table where all characters have the catcode "other" (12) except for space (10) assigned. So you get the same as your edef in TeX.

Answer 2

Using the token_filter callback and the token library (as Patrick kindly mentionned to me in comments to his answer) allows to build arbitrary token lists and insert them in the input stream. The callback should only be called once, it disables itself with callback.register(..., nil).

\expandafter\def\expandafter\B\expandafter{%
  \directlua
    {
      callback.register
        (
          'token_filter',
          function()
              callback.register ('token_filter', nil)
              return
                {
                  token.create(\number`\a, 11),
                  token.create(\number`\b, 11),
                  token.create(\number`\c, 12)
                }
          end
        )
    }%
  }
\show\B
\def\test#1#2#3{\show#1\show#2\show#3}
\expandafter\test\B

This answer only covers the Lua to TeX part. It should be possible to use token.expand() to do the TeX to Lua step, but I don't know how.

Answer 3

A quick summary/"cheat sheet" of the relevant commands (answer the question in title).

• TeX (convenience function): with \usepackage{luacode} it's possible to use \begin{luacode*} ... \end{luacode*} to write Lua code without worrying about catcode.

• Lua built-in has inspect() function used to print out some basic information about any (userdata) object / pretty-print tables etc..

• Tokens in Lua:

  • To get token from Lua: (see documentation for more details)

    • token.scan_toks(): get an argument (#1-style, must be wrapped in braces, drop outermost brace, can contain never-seen control sequences).

    • token.create("<control sequence name>"), token.create(int charcode, int catcode)

      Note create the impossible control sequence if it's never-seen

    • token.get_next() returns the next token in the input stream, must not be never-seen control sequence ^(*1)

    • token.new: ???

  • Token format:

    • check .csname

      • if is nil: explicit character

        • .mode = charcode
        • .cmdname ∈ (left_brace right_brace math_shift tab_mark mac_param sup_mark sub_mark spacer letter other_char).

      • else:

        • if .active: active character (charcode is in .csname, not .mode), else: control sequence

        • if .csname is the exact string "\csname\endcsname" and .tok = 0x20000000: is null control sequence token

        • else: is the active character/control sequence with the given csname.

        • check .cmdname for its meaning, can be one of the following

          token.commands() = {
           [0]="relax",
           "left_brace",
           "right_brace",
           "math_shift",
           "tab_mark",
           "car_ret",
           "mac_param",
           "sup_mark",
           "sub_mark",
           "endv",
           "spacer",
           "letter",
           "other_char",
           "par_end",
           "stop",
           "delim_num",
           "char_num",
           "math_char_num",
           "mark",
           "node",
           "xray",
           "make_box",
           "hmove",
           "vmove",
           "un_hbox",
           "un_vbox",
           "remove_item",
           "hskip",
           "vskip",
           "mskip",
           "kern",
           "mkern",
           "leader_ship",
           "halign",
           "valign",
           "no_align",
           "vrule",
           "hrule",
           "novrule",
           "nohrule",
           "insert",
           "vadjust",
           "ignore_spaces",
           "after_assignment",
           "after_group",
           "break_penalty",
           "start_par",
           "ital_corr",
           "accent",
           "math_accent",
           "discretionary",
           "eq_no",
           "left_right",
           "math_comp",
           "limit_switch",
           "above",
           "math_style",
           "math_choice",
           "non_script",
           "vcenter",
           "case_shift",
           "message",
           "normal",
           "extension",
           "option",
           "lua_function_call",
           "lua_bytecode_call",
           "lua_call",
           "in_stream",
           "begin_group",
           "end_group",
           "omit",
           "ex_space",
           "boundary",
           "radical",
           "super_sub_script",
           "no_super_sub_script",
           "math_shift_cs",
           "end_cs_name",
           "char_ghost",
           "assign_local_box",
           "char_given",
           "math_given",
           "xmath_given",
           "last_item",
           "toks_register",
           "assign_toks",
           "assign_int",
           "assign_attr",
           "assign_dimen",
           "assign_glue",
           "assign_mu_glue",
           "assign_font_dimen",
           "assign_font_int",
           "assign_hang_indent",
           "set_aux",
           "set_prev_graf",
           "set_page_dimen",
           "set_page_int",
           "set_box_dimen",
           "set_tex_shape",
           "set_etex_shape",
           "def_char_code",
           "def_del_code",
           "extdef_math_code",
           "extdef_del_code",
           "def_family",
           "set_math_param",
           "set_font",
           "def_font",
           "register",
           "assign_box_direction",
           "assign_box_dir",
           "assign_direction",
           "assign_dir",
           "combinetoks",
           "advance",
           "multiply",
           "divide",
           "prefix",
           "let",
           "shorthand_def",
           "def_lua_call",
           "read_to_cs",
           "def",
           "set_box",
           "hyph_data",
           "set_interaction",
           "letterspace_font",
           "expand_font",
           "copy_font",
           "set_font_id",
           "undefined_cs",
           "expand_after",
           "no_expand",
           "input",
           "lua_expandable_call",
           "lua_local_call",
           "if_test",
           "fi_or_else",
           "cs_name",
           "convert",
           "variable",
           "feedback",
           "the",
           "top_bot_mark",
           "call",
           "long_call",
           "outer_call",
           "long_outer_call",
           "end_template",
           "dont_expand",
           "glue_ref",
           "shape_ref",
           "box_ref",
           "data",
          }
          

          (call long_call outer_call long_outer_call) are user-defined macro.

  • To send back tokens from Lua to TeX:

    • tex.print(...): (with "trailing newline" i.e. on a separate line)

      • (token1)

      • { token1, token2 }

      • (string content)

      • (int catcodetablenumber, string content)

        (catcodetablenumber special values: -1 is current, -2 is detokenized)

      • ([int catcodetablenumber], {string1, string2, ...})

    • tex.sprint: same as above but without "trailing newline"

    • tex.tprint: shortcut

      tex.tprint(
          {int catcodetablenumber1, string content1},
          {int catcodetablenumber2, string content2},
          ...
      )
      

    • tex.cprint(int catcode, string content)

    • token.put_next( token1, token2 ), token.put_next{ token1, token2 }

      Note: you usually don't mix token.put_next and tex.print (print always come before all put_next!)

• Execute TeX code in Lua:

  • coroutines: print tokens to TeX and return control to TeX Concurrently interleaving execution of Lua and TeX in LuaTeX

  • tex.runtoks(function() tex.print(...) end): run the content immediately (dangerous! Do not change TeX mode inside the function)

• Execute Lua code in TeX:

  • \directlua{lua code to be executed... (x-expanded)}
  • Pass argument in with \luaescapestring{content... (x-expanded)}
  • Or leave the argument in the input stream and let Lua pick them up with e.g. token.scan_toks or token.get_next etc.

---

^(*1): otherwise it becomes \IMPOSSIBLE. A simple \futurelet can be used here safely, but there's certain overhead. Use scan_toks() if possible.

---

E.g. Lua function to increment the char code of each token by 1, works across all group levels and takes one expansion step. (demo for "take token list and do complex manipulation")

function f()
    local tokens=token.scan_toks()
    for i=1,#tokens do
        if tokens[i].cmdname=="letter" then
            tokens[i]=token.create(tokens[i].mode+1, 11)
        end
    end
    tokens[#tokens+1]=token.create(string.byte("}"), 2)
    token.put_next{
        token.create("pretty:n"),
        token.create(string.byte("{"), 1),
        table.unpack(tokens)
    }
end


\directlua{f()}{ab{c}123}  % eventually expand to bc{d}123

Another examples: https://tex.stackexchange.com/a/555222/250119