Tracing group-level and position in LuaTeX

Question

I am absolutely new to LuaTeX and I have few questions about it. First of all I learned that in LuaTeX it is possible to trace all TeX tokens with `token filter' callback. Is it possible to get some additional information in callback functions like current positions in input file and current group-level. What I am achieving to do is to write callback function which looks like:

\directlua{
  function foo ()
    t = token.get_next()
    if someTestFunction(t) then
       poz = currentPositionTexEyesAreLookingAtSourceFile()
       level = currentGroupLevel()
       mode = isMathMode()
       table.insert(someTable, {poz, level, mode})
    end
  end
  callback.register('token_filter',foo)
}

Answer 1

Yes, all these items are available from inside the token_filter callback. Some of them, however, require a bit of preparation.

Summary

To address the individual parts of your question:

• You can “trace” all tokens that pass TeX’s mouth using a combination of the token_filter and the token library. However, since this implies at least one call to the C API for each and every token, it is also quite inefficient. I doubt this approach beats \tracingall with respect to practicality so whatever you actually want to accomplish, it might not be what were looking for in the first place.

• The current nesting depth can be queried using the register tex.currentgrouplevel which corresponds to the eTeX primitive \currentgrouplevel.

• In order to test for math mode, TeX’s internal nesting stack (tex.nest) must be accessed. tex.nest.ptr contains the stack pointer so that tex.nest [tex.nest.ptr] returns the top of the stack (current state). If the field mathstyle is nonnegative then we are in fact in math mode. Additionally, inline and display math can be distinguished by checking whether the field mode is negative. (Theoretically the mode field alone should suffice for detecting math mode -- \ifmmode is implemented that way -- but it seems the values given in the reference are incorrect. Anyways, consult the source for further details.)

• Tracing input files and line numbers requires more effort. Since that kind of information is not directly available through the engine we have to manage it ourselves. Luckily, Luatex offers all the necessary prerequisites: for tracking the lines in the main input file you can use the process_input_buffer callback. Also, thanks to the open_read_line callback you have precise control over how TeX files are read. (The latter itself should be enough on its own when using a startup script. I didn’t test that, though.)

Example

Here is a demo that combines all of the above: https://gist.github.com/phi-gamma/6745931 . The result is a table with information about collected tokens:

The TeX file defines a couple user level macros. The first two control the callback state:

\enabletokenpos [<number>]  %% enable position tracking
\disabletokenpos            %% disable

Lines of the main file will be counted starting with the line where \enabletokenpos occurs. It takes one mandatory argument, namely the the current line number. To my knowledge the input line at that point cannot be automatically determined except by using a separate initialization script which would be outside the scope of the question at hand.

Then there are macros for accessing the current state:

\printcurrentfile      %% name of the file being read
\printcurrentpos       %% position in the file being read
\printcurrentfilepos   %% name and position in the customary “<name>:<line>” format

Obviously, the callbacks must be in place for them to work.

Tracing is controlled though these macros:

\collecttoken          %% add a single token to the tracing table
\startcollecttokens    %% start adding tokens (environment)
\stopcollecttokens     %% stop adding tokens

Caution when using the environment: due to multi-level expansion it will blow up the resulting table pretty quick. It is smart enough to not include itself, though.

Finally, in order to output the content of the token collection you need another macro:

\collectedtokens

It builds a simple table and prints it in the document.

Notes

Tracking files is restricted to input (i.e. TeX) files. Other files touched during a TeX rund -- Lua code, bitmaps, encodings, and the likes -- are unaffected. The table file_stack helps keeping track of the line numbers inside individual files. Both the process_input_buffer and the open_read_file callback are used. This is necessary in order to achieve a correct line count since the current implementation allows for the Lua code to be loaded while the document is being processed. As mentioned above, it should be possible to avoid this by using an initialization script to install the open_read_line filter before the main input is opened.

The function that is hooked into the token_filter comes in two variants: grabtoken() will consume a single token, add it to the collection, and remove itself from the callback. grabtokens() (plural) is basically the same function, but it will remain in place until it disabled by calling stopcollecting(). As a goodie, it will refrain from collecting any token after it encounters the control sequence \stopcollecttokens lest we pollute the collection with unwanted items.

Tokens are added as a table with the following keys:

• filename the name of the input file where the token was encountered in;

• position the line number of that input file;

• grouplevel value of tex.currentgrouplevel;

• mathmode positive if encountered inside display math, negative if in inline math, false otherwise;

• csname the associated control sequence, if appropriate;

• token the token table as returned by token.get_next(); the meanings of the values are explained in the reference.

E. g. \input looks like this:

{
  ["csname"]="input",
  ["file"]="tokenposition.tex",
  ["grouplevel"]=2,
  ["mathmode"]=false,
  ["position"]=32,
  ["token"]={ 119, 0, 3360 },
},