\lastnodetype not working as expected in luaTeX

Question

I'm currently experimenting with the linebreak_filter of luatex and when using it to change/replace the linebreaking algorithm the \lastnodetype command stopped working.

A MWE is the following:

\def\partest{%
   \directlua{%
     function hpackparagraph  (head) 
       return node.hpack(head)
     end
     callback.register("linebreak_filter", hpackparagraph)
  }%
  \par
}

\setbox0\vbox{ABC\par
  {\tracingonline1\showboxbreadth\maxdimen\showboxdepth\maxdimen\showlists}%
  \showthe\lastnodetype
}

\setbox0\vbox{ABC\partest
  {\tracingonline1\showboxbreadth\maxdimen\showboxdepth\maxdimen\showlists}%
  \showthe\lastnodetype
}

In a nutshell the \partest command changes the paragraph breaking algorithm to simply pack the unbroken paragraph into an hbox and return that hbox unchanged to the vertical list (not useful standalone, but this is the shortest version that shows my problem).

So if you run this you get:

This is LuaTeX, Version beta-0.70.1-2011061421 (rev 4277) 
 restricted \write18 enabled.
(./lua-bug.tex

### internal vertical mode entered at line 13
\hbox(6.83331+0.0)x469.75499, glue set 427.9494fil, direction TLT
.\whatsit
..\localinterlinepenalty=0
..\localbrokenpenalty=0
..\localleftbox=null
..\localrightbox=null
.\hbox(0.0+0.0)x20.0, direction TLT
.\tenrm A
.\tenrm B
.\tenrm C
.\penalty 10000
.\glue(\parfillskip) 0.0 plus 1.0fil
.\glue(\rightskip) 0.0
prevdepth 0.0, prevgraf 1 line
### vertical mode entered at line 0
prevdepth ignored

! OK.
l.14 ...h\maxdimen\showboxdepth\maxdimen\showlists
                                                  }%
? 
> 1.
l.15   \showthe\lastnodetype

? 

### internal vertical mode entered at line 18
\hbox(6.83331+0.0)x41.8056, direction TLT
.\whatsit
..\localinterlinepenalty=0
..\localbrokenpenalty=0
..\localleftbox=null
..\localrightbox=null
.\hbox(0.0+0.0)x20.0, direction TLT
.\tenrm A
.\tenrm B
.\tenrm C
.\penalty 10000
.\glue(\parfillskip) 0.0 plus 1.0fil
prevdepth ignored
### vertical mode entered at line 0
prevdepth ignored

! OK.
l.19 ...h\maxdimen\showboxdepth\maxdimen\showlists
                                                  }%
? 
> -1.
l.20   \showthe\lastnodetype

As you can see the \lastnodetype for the first test is 1(hbox) as expected. But for the second it is -1 which means current list has no nodes.

This of course is not correct as it does have an hbox inside the one returned by my primitive linebreak_filter. The only difference between the two lists that one can observe is that in the first one the hbox is 469pt wide while in the second one it is at its natural width, but this is expected as it was packed this way.

My guess is that the node list that is placed by the standard paragraph builder is updating the data structure for \lastnodetype but that doesn't happen is such a filter is applied which feels like a bug. Or am I missing something fundamental here?

Update

Consistent with the problem the command \lastbox also returns a void box in the case where the linebreak_filter is applied, i.e., \setbox0\lastbox \showbox0 fails to return the last object in the current list.

Answer 1

I'm now fairly certain this this is a bug in the current implementation of linebreak_filter. What seems to happen is the following: when TeX builds a list (in this case a vertical list) it keeps track of it through a pointer to the head and to the tail of that list. Now when a paragraph is broken into lines those lines get appended to the current vertical list by the linebreaking algorithm and the tail pointer is updated to point to the new end of this list. However, if the linebreak_filter replaces TeX's algorithm the material from that filter is appended to the list but the tail pointer is not changed. Thus in my example it still points to the node before the paragraph (which happens to be nil as the paragraph starts the vbox).

Fortunately this tail pointer is accessible and modifiable from Lua code, so there is a way to fix this. We can't do this in linebreak_filter, however, because the current luatex code that calls the filter expects the tail in precisely this wrong as it does some final checkup and changes.

But what we can do is to use the followup file post_linebreak_filter to do the work for us, because by then all these things have happened. So the workaround looks like this:

function hpack_paragraph  (head)
  h =  node.hpack(head)
  return h
end

function fix_nest_tail (head)
  tex.nest[tex.nest.ptr].tail = node.tail(head)
  return true
end

callback.register("linebreak_filter", hpack_paragraph)
callback.register("post_linebreak_filter",fix_nest_tail)

The second filter simply updates TeX's tail pointer in the current semantic nest to become the tail of the result produced from the line breaking. A bit nasty, but at least it allows to use the linebreak_filter in the current implementation.

If you wonder what this this code does: without anything further it changes the output so that every paragraph is returned as a single line (and indentations like from itemize or similar environments are dropped) and inserts like footnotes or \vadjusts stay within those hboxes. So it needs a little more to become useful, but let your imagination run.

Update

Hans Hagen pointed out to me that a proper linebreak_filter would normally need to set a few more variables, namely things like prevdepth and prevgraf and possibly add glue nodes representing a baselineskip. In contrast to the issue with the tail pointer these could be set in the filter directly, so a more complete solution would look then like this:

function oneliner(head)
    local h = node.hpack(head)
    local d = tex.baselineskip.width - tex.nest[tex.nest.ptr].prevdepth - h.height
    tex.nest[tex.nest.ptr].prevdepth = h.depth
    tex.nest[tex.nest.ptr].prevgraf  = 1
    local n
    if d < tex.lineskiplimit then
        n = 1
        d = tex.lineskip
    else
        n = 2
    end
    local s = node.new("glue_spec")
    local n = node.new("glue",n)
    s.width = d
    n.spec = s
    return node.insert_before(h,h,n)
end

callback.register("linebreak_filter",oneliner)

-- fix the bug with the tail pointer not being updated:

function fix_nest_tail (head)
  tex.nest[tex.nest.ptr].tail = node.tail(head)
  return true
end

callback.register("post_linebreak_filter",fix_nest_tail)

For the use case that I had in mind this extra coding is not necessary, as I want to store away those boxes anyway for later use, but in case one wants to write a real line breaking algorithm this kind of additional work definitely needed, so it makes sense to remark on this here.