LuaTeX hpack_filter callback not triggering on certain hlist creations

Question

EDIT: The hpack_filter is triggered and my question pointless. Consult my answer on why I missed the callback invocation.

I am using LuaTeX to analyze the creation process of math expressions in a document. To this effect, I use the mlist_to_hlist callback to insert a whatsit node into the node list. When the math expressions are assembled into their final form by packing various node lists into horizontal and vertical lists, I hook into this process with hpack_filter and vpack_filter callbacks. This allows me to follow the creation path from the raw math expression all the way to the final page.

I encounter an issue with a specific part of this process. It seems that the rows of an align-math-environment, are added to the main vertical list as horizontal lists with a specific subtype. These hlist nodes have subtype 4, which is described as 'alignment column or row' in the LuaTeX documentation.

My problem now is that the creation of exactly these hlists is not filtered with the hpack_filter callback. For all other hlist types I get the expected invocation of hpack_filter but not for these. For my purposes, it already suffices to check the input arguments of this process for the occurrence of certain whatsit node and then completely cancel the packing. The goal is to not have this content transferred to the main vertical list.

This leads to my questions:

• Why is hpack_filter not called for the creation of an hlist with subtype 4?
• Is it possible to enable such a behavior or to somehow obtain an equivalent callback mechanism by other means?

Answer 1

While inspecting the node list that were given as arguments to the hpack_filter callbacks, I did not notice that the input argument contained unset nodes. My take on how an alignment (that is generated by the TeX command \halign) is treated by LuaTeX is as follows:

1. Through a series of hpack invocation with argument groupcode == align_set, the input node lists are packed into the head field of unset nodes.
2. These unset nodes (with glue nodes and perhaps other stuff) are then packed into an hlist node with subtype 4. The node lists that resided in the head fields of the unset nodes are still present but now in head fields of hlist nodes with subtype 5. This packing happens in an hpack invocation with argument groupcode == fin_row. This exactly the part that I missed and that I was asking for.

This is what I figured out by looking at intermediate node lists and is my own opinion. This information cannot be found in LuaTeX manual so far (version 0.76) and unset nodes are hardly mentioned at all. That they have a head field I deduced from looking at the LuaTeX source code.