Why do user-defined whatsit nodes affect LuaTeX pagebreaking?

Question

EDIT: A summary of the possible answers is given below.

In my current LuaTeX project I am using user-defined whatsit nodes to transfer information between the different stages of a LuaTeX run. On this topic, the manual states the following:

The LUATEX engine will simply step over such whatsits without ever looking at the contents.

My interpretation of this is, that I can add user-defined whatsit nodes and nothing should change in the actual processing of the node lists (i.e. the document should look the same with or without them).

I am encountering the following problem, however. In my case, a certain paragraph, let's call it P, would overflow the page and is moved to the next page. Accordingly, the white space between the remaining paragraphs on the previous page is stretched to fill the page (resp. column).

If I add a user-defined whatsit node as first node of P, the content of the paragraph is moved to the next page (as before), BUT the whatsit node STAYS on the previous page. Subsequently, whitespace is added at the bottom of the last column on the previous page, the ensure correct inter-paragraph spacing.

In my opinion, this is not the correct behavior and I have the following two questions:

1. Can I force the whatsit node to stay with its paragraph? (But preferably not by packing it into additional horizontal or vertical lists.)
2. If not, can I at least suppress the generation of inter-paragraph white space at the bottom of the column?

Additional Information

An illustration of how it looks like:

         Without whatsit                  With whatsit

Page n  |text          |        Page n  |text          |
        |text          |                |              |
        |              |                |text          |
        |text          |                |text          |    Incorrect
        |text__________|                |______________| <- inter-paragraph
                                                            glue
         ______________                  ______________
Page n+1|text of P     |        Page n+1|text of P     |
        |text of P     |                |text of P     |
        |              |                |              |
        |text          |                |text          |

---

In detail I do the following:

With a post_linebreak_filter callback, I add my user-defined whatsit node as new head of the vertical paragraph list (by using node.insert_before(head, head, my_whatsit)). This node is still visible at the shipout stage, where it gets integrated into the vertical list of the current column. In the case of a paragraph that gets moved to the next page (as in my example above), the whatsit node can be found in the vertical column list of page n, while the rest of the paragraph is moved to page n+1. I would need it to stay with the paragraph.

---

Answers

Pure LaTeX

The answer of Dan explains how to dock the whatsit to the paragraph by the use of \penalty and \nobreak.

LuaTeX

David Carlisle and topskip gave two different ways to achieve this behavior by altering the insertion location of the whatsit node in the post_linebreak_filter callback. topskip's solution recursively searches for the first hlist in the paragraph and inserts the whatsit node into it. David's approach aims to insert the whatsit node after the first node (usually a glue node) and thus anchors it to the paragraph. In my case this can be achieved by simply changing node.insert_before(head, head, my_whatsit) to node.insert_after(...). This keeps the whatsit node in the same vlist as the paragraph's hlists, which is the reason why I selected this as my solution. As David already mentioned, it would be better to insert the whatsit node after all discardable nodes (and not just after the first one).

Answer 1

Stealing gratuitously from topskip's answer (although I did vote for it as well:-) He showed how to push the whatsit into the horizontal list (which is basically what \textcolor{}.. does as opposed to \color) But if you want the whatsit to be in the vertical list but have no break point before the following paragraph, you can do as I suggest in comments and add it further down the list.

\documentclass{article}
\usepackage{luacode,luatexbase}
\begin{document}

jj

\begin{luacode*}
w = node.new("whatsit","user_defined")
w.type=100

function foo(head)
  new_head = node.insert_before(head, head.next, w)
  return new_head
end

luatexbase.add_to_callback("post_linebreak_filter",foo,"foo")    
\end{luacode*}

Hello 
\showoutput

\end{document}

the only difference from topskip's (guess of your) code is that I changed head to head.next to move the whatsit after the lineskip glue. (more careful code could test the node type and iterate until it found the first non-discardable node, but this demonstrates the effect. If you compare the log files of the version with or without .next you see

*** 143,150 ****
  ....\glue(\parfillskip) 0.0 plus 1.0fil
  ....\glue(\rightskip) 0.0
  ...\glue(\parskip) 0.0 plus 1.0
- ...\glue(\baselineskip) 3.11111
  ...\whatsit0=0
  ...\hbox(6.94444+0.0)x345.0, glue set 307.49994fil, direction TLT
  ....\whatsit
  .....\localinterlinepenalty=0
--- 143,150 ----
  ....\glue(\parfillskip) 0.0 plus 1.0fil
  ....\glue(\rightskip) 0.0
  ...\glue(\parskip) 0.0 plus 1.0
  ...\whatsit0=0
+ ...\glue(\baselineskip) 3.11111
  ...\hbox(6.94444+0.0)x345.0, glue set 307.49994fil, direction TLT
  ....\whatsit
  .....\localinterlinepenalty=0

which says, if you are not used to unix diff format that the lines

  ...\whatsit0=0
 ...\glue(\baselineskip) 3.11111

have swapped order, showing that the baselineskip glue (which is where the page break happened in your example) now occurs before the whatsit.

Answer 2

A whatsit (user-defined or not) is always non-discardable. So if glue follows it, that glue constitutes a break point. If glue precedes it, that might stretch along with the rest of the page. To make it adhere to the following paragraph, you can make it part of the paragraph. The LaTeX way is to put \leavevmode before it. That positions the whatsit at the baseline of the first line of that paragraph. If that positioning makes a difference, and you want it to be in vertical mode just before the paragraph, you can put a slightly more favorable break before the whatsit and/or a \nobreak after it.

\par
\penalty 10
<my whatsit>
\nobreak
<start of paragraph>

Documents do not always paginate the same, nor do paragraph always break the same when whatsits are included. Well-know examples are color specials. If yous say \color{red} just before a paragraph starts, then that \special occurs just before the \parskip glue. This can cause a pagebreak with the color special at the bottom of the previous page and the paragraph on the next.

Another well-known example where a whatsit causes a change of position: a color special at the start of a parbox with [t] option. The baseline of that parbox is then at that \special rather than at the base of first text line.

Answer 3

You should add the whatsit into paragraph itself (i.e. the horizontal list).

I guess you have something like this:

\documentclass{article}
\usepackage{luacode,luatexbase}
\begin{document}

\begin{luacode*}
w = node.new("whatsit","user_defined")
w.type=100

function foo(head)
  new_head = node.insert_before(head, head, w)
  return new_head
end

luatexbase.add_to_callback("post_linebreak_filter",foo,"foo")    
\end{luacode*}

Hello 

\end{document}

which results into the whatsit inside the vertical list (as the others have written). See this representation:

Perhaps you should (I don't really know what your whatsit is for) add it to the paragraph itself. You can do this for example:

\documentclass{article}
\usepackage{luacode,luatexbase}
\begin{document}

\begin{luacode*}
w = node.new("whatsit","user_defined")
w.type=100

function foo(head)
  while head do
    if head.id == 0 then
      -- hlist
      node.insert_after(head.list,head.list,w)
      return true
    end
    if head.id == 1 then
      -- vlist
      foo(head.list)
    end
    head = head.next
  end
end

Hello
luatexbase.add_to_callback("post_linebreak_filter",foo,"foo")
\end{luacode*}

\end{document}

which gives

And now the whatsit is part of the hlist.