How do I tell a node about LuaTeX's state of mind when it made it?

Question

To be absolutely clear before I begin, this is a question about LuaTeX and so "node" in this question means LuaTeX's notion of a node, not any other TeX package's notion.

Ultimately, I want to detect when TeX changes group and record that information in the node list so that when I come to process that node list later on (via pre_linebreak_filter) then I can figure out the group level that existed when each node was created.

My existing method of doing this was to use an attribute which was set to tex.currentgrouplevel. However, as attributes are static, I needed to reset that every time TeX changed group. There doesn't seem to be a callback for that specific event, so instead I used token_filter to set it every time a token was read. It's not very efficient, but I only wanted it for short segments at a time so I would install the callback when I wanted it and remove it again when done.

Specifically, my lua code for the callback was:

local function setlevel ()
   tex.attribute[1] = tex.currentgrouplevel
   return token.get_next()
end

where I'd already set up the attribute in TeX.

However, the current version of LuaTeX has removed the token_filter callback so this no longer works! The recommendation appears to be to use the functionality of the token library, specifically the scanner. However, the examples of this that I have seen imply that it has to be called explicitly. So in order to examine each token as it was read to see if it is makes a group change, I'd have to keep reinvoking the scanner and it would appear that this can only be done by inserting tokens into the input stream. This gets tricky because some tokens may need processing by TeX's mouth and stomach and insert stuff back in to the input stream.

An alternative strategy would be to adjust the node creation code. However, my experiments (admittedly with an earlier version of LuaTeX) with changing the node.new function show that this is not called when LuaTeX creates a node for itself but only when user-written code is run (I guess this makes sense for implementation reasons).

So, is there a way to achieve what I want with the current version of LuaTeX? Namely, that each created node knows the TeX group level at which it was created.

(In the "Similar Questions" was Tracing group-level and position in LuaTeX which as a question would be a duplicate, but the accepted answer to that suffers from the same issue as it uses token_filter.)

Answer 1

---

math

---

For nodes generated from math node such as the a b^2 / {a b}^2 example in comments, the situation is easier as a {} group maps to a horizontal list just like \hbox and so is directly visible in the node tree.

taking that example

\documentclass{article}


\begin{document}


\directlua{
function foo(n,s)
  print "\string\n callback\string\n"
  local typ=node.types()
  local nn=n
  while(nn \string~= nil) do
    print(typ[nn.id] ..'(' .. nn.id .. ')')
    if(nn.id==0 and nn.list \string~= nil) then
      local nnn=nn.list
      while(nnn \string~= nil) do
        print('..' .. typ[nnn.id] ..'(' .. nnn.id .. ')')
        nnn=nnn.next
      end
    end
    nn=nn.next
  end
  return n
end
luatexbase.add_to_callback("pre_linebreak_filter",foo,"foo")
}
$ a b ^2$

${a b}^2$

\end{document}

so the a b ^2 produces

math(11)
glyph(29)
glyph(29)
hlist(0)
..glyph(29)
math(11)

so a math start then a and b at the top level and a nested list for the superscript which has the glyph 2, then the math end.

Conversely {a b}^2 produces

math(11)
hlist(0)
..glyph(29)
..glyph(29)
hlist(0)
..glyph(29)
math(11)

so that's a math start then a nested hlist corresponding to the {} containing the glyphs a and b then another hlist corresponding to the superscript, containing the list 2, then the math end.

---

text

---

To deal with text mode you need to try a bit harder, this uses the input buffer callback to insert \zz at every { it's important that \zz expands to nothing as it will be inserted in some tricky places eg \begin{itemize} becomes \begin{\zz itemize} which still works...

In Lua code, \zz sets the \grpdepth attribute to the current grouping depth.

Finally we pick these up as above in the pre_linebreak callback just to debug the constructed node list.

The input

{{{{{{{{{a{b{c{d{e}}}}}}}}}}}}}

produces a console output

..glyph a[depth=9]
..glyph b[depth=10]
..kern  [depth=-1]
..glyph c[depth=11]
..glyph d[depth=12]
..glyph e[depth=13]

The full test document is

\documentclass{article}

\newattribute\grpdepth
\edef\grpdepthnum{\the\allocationnumber}

\def\zz{\directlua{%
tex.attribute.grpdepth=tex.currentgrouplevel
print('\string\nCURRENT LEVEL = ' .. tex.attribute.grpdepth)
}}

\def\startattributing{\directlua{%
function flaggroup (s)
 return string.gsub(s,'{','{\string\\zz ')
end
  local zzz='}}'
luatexbase.add_to_callback(
  'process_input_buffer',
  flaggroup,
  'flaggroup')
}}

\def\endattributing{\directlua{%
luatexbase.remove_from_callback(
  'process_input_buffer',
  'flaggroup')
}}


\directlua{
function foo(n,s)
  print "\string\n callback\string\n"
  local typ=node.types()
  local nn=n
  while(nn \string~= nil) do
        na=node.get_attribute(nn,\grpdepthnum) or -1
        print('..' .. 
              typ[nn.id] .. ' ' ..
              string.char(nn.char or 32) ..
              '[depth=' .. na .. ']'
              )
    if(nn.id==0 and nn.list \string~= nil) then
      local nnn=nn.list
      while(nnn \string~= nil) do
        na=node.get_attribute(nnn,\grpdepthnum) or -1
        print('..' .. 
              typ[nnn.id] .. ' ' ..
              string.char(nnn.char or 32) ..
              '[depth=' .. na .. ']'
              )
        nnn=nnn.next
      end
    end
    nn=nn.next
  end
  return n
end
luatexbase.add_to_callback("pre_linebreak_filter",foo,"foo")
}
\begin{document}


\startattributing

\def\f{a}\meaning\f

aaa \emph{bbb}  \mbox{jjj} {ccc ddd} {{{x}}}

{{{{{{{{{a{b{c{d{e}}}}}}}}}}}}}

\begin{itemize}
\item zzz
\end{itemize}

\endattributing

\end{document}