Can the Lua part of LuaTeX know about tokens?

Question

Using \directlua and tex.print, it is possible to send strings of characters back and forth between TeX and Lua. Let's take an example: starting with a macro \A, I pass its replacement text to Lua and back, defining \B

\edef\A{ab\string c}
\directlua{tex.print("\noexpand\\def\noexpand\\B{\unexpanded\expandafter{\A}}")}
\show\A
\show\B
\ifx\A\B
  \message{Identical :)!}
\else
  \message{Different :(.}
\fi
\bye

The two macros appear identical, but they aren't: the weird/special combination of category codes (letter and other) is lost before reaching Lua, and tokenization on the return trip is done using the category code régime in effect when \directlua is performed.

Can the \directlua line be replaced by some other Lua code which would be able to define \B to be identical to \A, including category codes? Of course,

\directlua{\unexpanded{tex.print("\\edef\\B{\\unexpanded\\expandafter{\\A}}")}}

would work (by delaying the expansion of \A until after the passage through LuaTeX), but my goal would be to perform some complicated operations on the token lists on the Lua side.

Answer 1

You can play with tex.print() and catcode tables. The result of

tex.tprint({"\\def\\B{ab"},{-2,"c"},{"}"})

gives "Identical :)". This is, because -2 is a predefined catcode table where all characters have the catcode "other" (12) except for space (10) assigned. So you get the same as your edef in TeX.

Answer 2

Using the token_filter callback and the token library (as Patrick kindly mentionned to me in comments to his answer) allows to build arbitrary token lists and insert them in the input stream. The callback should only be called once, it disables itself with callback.register(..., nil).

\expandafter\def\expandafter\B\expandafter{%
  \directlua
    {
      callback.register
        (
          'token_filter',
          function()
              callback.register ('token_filter', nil)
              return
                {
                  token.create(\number`\a, 11),
                  token.create(\number`\b, 11),
                  token.create(\number`\c, 12)
                }
          end
        )
    }%
  }
\show\B
\def\test#1#2#3{\show#1\show#2\show#3}
\expandafter\test\B

This answer only covers the Lua to TeX part. It should be possible to use token.expand() to do the TeX to Lua step, but I don't know how.