As seen from answers to my previous question, math boundary nodes are not a reliable way of marking math boundaries. There are two possible workarounds that can be used to mark boundaries: 1) Add userdefined whatsit nodes around math list, 2) Define and set a "math" attribute for nodes inside math list. Such marking of boundary can be handy in many ways to post process nodelist after TeX is done breaking lines: a) add <beginmath> <endmath> type delimiters while trying to extract text b) selective coloring of text and math, c) etc. a) can also be used alongside "alternative text" added to beginmath whatsit value to provide alternative/text description of equation.

Concerning approach 1), LuaTeX document states "The LuaTEX engine will simply step over such whatsits without ever looking at the contents." Though there are posts that suggest that whatsits can affect microtypography, linebreaking, and pagebreaking output: link to a post that shows affected page breaking, and link to post that shows affected microtypography (which can also affect linebreaking in TeX).

Given approach 2) requires iterating over all math list contents, it would be slower compared to 1) if called using mlist_to_hlist filter? (Unless probably $<tex_equation>$ can be safely redefined to set and unset an attribute before & after $ delimiter, though there are more such delimiters...). So the question is how can one achieve marking the math boundary using approach 1) such that it truly does not affect microtypography, linebreaking, and pagebreaking? Or is approach 2) the only safe bet?

Here's my first try for inline math, am looking for expert review, and extension for display math (align, etc environments).


% >>> lualatex mathmode.tex


local function mathboundary(h,d,p)
    local beginmath = node.new("whatsit","user_defined")
    beginmath.type = string.byte("s")
    beginmath.value = "beginmath"
    beginmath.attr = h.attr
    beginmath.user_id = luatexbase.new_whatsit'beginmath'
    local endmath = node.new("whatsit","user_defined")
    endmath.type = string.byte("s")
    endmath.value = "endmath"
    endmath.attr = h.attr
    endmath.user_id = luatexbase.new_whatsit'endmath'
    h = node.insert_after(h,node.tail(h),endmath)
    h = node.insert_before(h,h,beginmath)
    return node.mlist_to_hlist(h,d,p)

luatexbase.add_to_callback('mlist_to_hlist', mathboundary, 'Mark math')

\setbox0=\vbox{{\noindent Hello\\ $x=a+b^2$}}

    local nodetree = require('nodetree')



Console (look for WHATSIT subtype: user_defined):

└─VLIST width: 345pt, depth: 0.83pt, height: 18.94pt
    ├─HLIST subtype: line, width: 345pt, depth: 0.11pt, height: 6.94pt
    │ ╚═head:
    │   ├─LOCAL_PAR 
    │   ├─GLYPH subtype: 256, char: H, width: 7.5pt, height: 6.83pt
    │   ├─GLYPH subtype: 256, char: e, width: 4.44pt, height: 4.48pt, depth: 0.11pt
    │   ├─GLYPH subtype: 256, char: l, width: 2.78pt, height: 6.94pt
    │   ├─GLYPH subtype: 256, char: l, width: 2.78pt, height: 6.94pt
    │   ├─GLYPH subtype: 256, char: o, width: 5pt, height: 4.48pt, depth: 0.11pt
    │   ├─PENALTY penalty: 10000
    │   ├─GLUE stretch: +1fil
    │   ├─PENALTY penalty: -10000
    │   └─GLUE subtype: rightskip
    ├─PENALTY subtype: linebreakpenalty, penalty: 300
    ├─GLUE subtype: baselineskip, width: 3.75pt
    └─HLIST subtype: line, width: 345pt, depth: 0.83pt, height: 8.14pt
        ├─WHATSIT subtype: user_defined, user_id: 1, type: 115, value: beginmath
        ├─GLYPH subtype: 256, char: x, width: 5.72pt, height: 4.31pt
        ├─GLUE subtype: thickmuskip, width: 2.78pt, stretch: 2.78pt
        ├─GLYPH subtype: 256, char: =, width: 7.78pt, height: 3.67pt, depth: -1.33pt
        ├─PENALTY subtype: noadpenalty, penalty: 500
        ├─GLUE subtype: thickmuskip, width: 2.78pt, stretch: 2.78pt
        ├─GLYPH subtype: 256, char: a, width: 5.29pt, height: 4.31pt
        ├─GLUE subtype: medmuskip, width: 2.22pt, stretch: 1.11pt, shrink: 2.22pt
        ├─GLYPH subtype: 256, char: +, width: 7.78pt, height: 5.83pt, depth: 0.83pt
        ├─PENALTY subtype: noadpenalty, penalty: 700
        ├─GLUE subtype: medmuskip, width: 2.22pt, stretch: 1.11pt, shrink: 2.22pt
        ├─GLYPH subtype: 256, char: b, width: 4.29pt, height: 6.94pt
        ├─HLIST subtype: sup, width: 4.49pt, height: 4.51pt, shift: -3.63pt
        │ ╚═head:
        │   └─GLYPH char: 2, width: 3.99pt, height: 4.51pt
        ├─WHATSIT subtype: user_defined, user_id: 2, type: 115, value: endmath
        ├─MATH subtype: endmath
        ├─PENALTY subtype: linepenalty, penalty: 10000
        ├─GLUE subtype: parfillskip, stretch: +1fil
        └─GLUE subtype: rightskip

  • Please add some information about what you want the marks to be used for. Nov 21, 2020 at 7:53
  • @MarcelKrüger Done, though IMHO adding boundary by itself should be orthogonal to its intended use. Put another way, am looking for a generic enough solution that I code a method to add boundary around inline and display math, forget about it, and just focus on using it in myriad ways in future.
    – codepoet
    Nov 21, 2020 at 8:06
  • @MarcelKrüger And updated again with another use case: "a) can also be used alongside "alternative text" added to beginmath whatsit string value to provide alternative/text description of equation."
    – codepoet
    Nov 21, 2020 at 8:21

1 Answer 1


When looking at nodelists after linebreaking, boundary nodes are not a reliable solution. This has at least two reasons:

  1. Every node is either discardable or not discardable. If your boundary node is discardable, you get the same problem as with math-on/math-off nodes: They disappear at the start of a line. So the node you want can not be discardable, but remember the rule about discardable nodes at the beginning of a line: After a linebreak, all discardable nodes are dropped until the first non-discardable node is encountered. Especially this implies that your new non-discardable boundary node would suppress the removal of potentially following discardable ones, affecting the output. (There are other reasons why adding a non-discardable node here, e.g. a whatsit, will always affect linebreaking, but this one is the easiest to describe)

  2. Once linebreaking comes along, you can have hlist containing math but not containing any math boundary. A rather extreme example would be:

      Hi $1+2+3+4$
    \setbox1\vsplit0 to\baselineskip 
    \setbox2\vsplit0 to\baselineskip

    Here box 1 ends up containing only a middle part of a math formula, so there is no reason for it containing any kind of boundary node. It is still math though.

    (A variant of this which is more likely to happen in practise is a math formula breaking across pages. The second page will not have a start math node because no math starts there.)

So what can be done about this? There are two options:

  1. While your approach 1) can't work reliably, you can implement 2) and add an attribute. You can basically just iterate over the nodelist in post_mlist_to_hlist_filter¹ to add this, then both inline and display math will be covered.
  2. Do whatever you want to do before linebreaking and just use the existing math boundary nodes. In most cases, you want to do some work before linebreaking anyway so this is quite natural.

Depending on the specific use-case, one of the solutions will often be quite natural. E.g. text extraction should almost always be based on pre-linebreak (or even pre-shaping) text anyway, so it fits good to 2, while adding different colors for math nodes effectivly just means adding an attribute anyway, so it fits naturally to 1.

¹: In modern LuaLaTeX versions, mlist_to_hlist should almost never to set directly but instead pre_mlist_to_hlist_filter or post_mlist_to_hlist_filter should be used. They have basically the same interface as mlist_to_hlist, but you don't have to call node.mlist_to_hlist yourself and your code stays compatible with other packages using these callbacks. The code in your question for example could be adjusted by replacing

luatexbase.add_to_callback('mlist_to_hlist', mathboundary, 'Mark math')


luatexbase.add_to_callback('pre_mlist_to_hlist_filter', mathboundary, 'Mark math')

and replacing

return node.mlist_to_hlist(h,d,p)

with just

return h
  • Thanks! For "text extraction should almost always be based on pre-linebreak (or even pre-shaping) text anyway", I was thinking of extracting text that falls on a page, so that has to be post linebreak. So it won't work to use math nodes pre linebreak.
    – codepoet
    Nov 21, 2020 at 10:04
  • @reportaman I think it depends, and it's worth thinking about your requirements at a more conceptual level. In general, line-breaking / page-breaking are transformations that could in principle radically alter the text. For example, line-breaking often introduces hyphens, but it could in principle insert a lot more. Then the question becomes: do we want the portion of our input text before line-breaking (in which case do text-extraction pre-linebreak, but somehow—like SyncTeX—track which text went onto which page), or do we want whatever arbitrary text ended up on the page after linebreaking? Nov 21, 2020 at 20:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .