luatex node library: iterate over whatsit save_pos to get the position of linebreaks

Question

Marcel's answer resolved the original question. I have added followup questions from his answer under my original question. Followup questions begin with a similar block quote as this stating "Followup questions begin here:"

Am trying to learn luatex node library, and here's what I thought I would want to do as an experiment:

1. Append a whatsit save_pos node to every line of text.
2. Iterate over all the whatsis save_pos nodes to retrieve position of linebreaks/lineendings and either save it to some array to use at the end of page or beginning of next page.

I know how to do 1., though don't know how to do 2 (could not decipher a way to do that from luatex manual).

AFAIK, positions on page are only known after a certain point in the processing of page, so I cannot probably use a linebreak callback right away to know that information. Though I do not know how to traverse all the elements of the page (at the end of page building process, or of previous page at the beginning of next page) to get a list of positions of all linebreaks. Ideally I would like to traverse all elements on the page as save_pos could be placed anywhere (inside hbox, inside vbox, in main vertical list, header/footers, or textpos overlays, etc).

Here's my code so far:

\documentclass[notitlepage,letterpaper]{article}

%\usepackage{lua-visual-debug}
\usepackage[absolute]{textpos}
\usepackage[letterpaper,left=0in,right=0in,top=1in,bottom=1in]{geometry}
\usepackage[expansion=alltext,shrink=20,stretch=20]{microtype}
\usepackage{fontspec}
\usepackage{blindtext}
%\setmainfont{Verdana} % Commented this as standard linux installs do not come with Verdana, check Marcel's comment to this question.

\directlua{
    function my_post_lb_filter(head,groupcode)
      local HLIST = node.id("hlist") % node.id for a line of text in vertical list
      for n in node.traverse(head) do % For every subnode within node head
        if n.id==HLIST then % If its a line of text
          local savepos = node.new("whatsit","save_pos")
          n.head = node.insert_after(n.head,node.tail(node.list),savepos)
        end
      end
      return head
    end
  luatexbase.add_to_callback('post_linebreak_filter', my_post_lb_filter, 'Play with luatex node library')
}


\begin{document}

\raggedright
\newbox\myoddvbox
\newbox\myoddvboxsplit
\global\setbox\myoddvbox=\vbox{{\hsize=2in \blindtext[1]\endgraf}}%
\setbox\myoddvboxsplit=\vsplit\myoddvbox to 4cm


\begin{textblock*}{0.5\linewidth}[0,0](2in,2in)
\begin{minipage}[t][1cm][t]{0.5\linewidth}%
  \box\myoddvboxsplit%
\end{minipage}
\end{textblock*}


\begin{textblock*}{0.5\linewidth}[0,0](5in,2in)
\begin{minipage}[t][1cm][t]{0.5\linewidth}%
  \box\myoddvbox%
\end{minipage}
\end{textblock*}


\null
\newpage

% replace following line with a iterate function to print all the positions from lastpage
\the\lastxpos, \the\lastypos.


\end{document}

Followup questions begin here:

1. What is the sequence of execution of late_lua nodes? Is it a FIFO model?
2. What is the nature of data that can be acted on? Can we access and late process hlist node (for instance insert_to) from a latelua function? Or is it too late for that? What is the life of a hlist node anyway, is it freed after a page is shipped out?
3. Can we pass arguments to a late_lua function? I tried passing arguments to the function savepos.data = savepos_func("hello"), and the lua segfaults.
4. luatex manual says the type of data field of late_lua node can be either string or function. When would one assign a string to a late_lua node's data field? I tried assigning a lua string savepos.data = "hello world", and it gave a error during shipout [\latelua]:1: syntax error near 'world'.
5. What is the scope of code that goes in directlua? I tried to print the size of table position_array using \AtBeginShipout of package \usepackage{atbegshi}, and it prints nil on terminal (code is simple print(tostring(table.getn(position_array))) in a \directlua macro). If I remove local from the declaration local position_array = {}, then it prints the correct value of the array (while processing next page).
   • Why did you not select function getpos_and_reset to be local, while making function getpos_and_reset, and table position_array local?

Answer 1

If you are using Lua code, you basically never need save_pos whatsits. save_pos whatsits are the whatsits inserted by pdfTeX's \savepos primitive. Only the position of the last save_pos is saved, so if you don't run code between two of these whatsits the first one is basically useless. (The typical use for \pdfsavepos is in combination with \write. Using code like \pdfsavepos\write42{(\the\pdflastxpos,\the\pdflastypos)} it allows to write a position to some file.) While you could use this with late_lua callback to get the current coordinates, that is already available without save_pos through pdf.getpos.

So to get an array of all linebreaks, the easiest way is to directly add late_lua nodes which save the linebreak locations during shipout:

\documentclass[notitlepage,letterpaper]{article}

%\usepackage{lua-visual-debug}
\usepackage[absolute]{textpos}
\usepackage[letterpaper,left=0in,right=0in,top=1in,bottom=1in]{geometry}
\usepackage[expansion=alltext,shrink=20,stretch=20]{microtype}
\usepackage{blindtext}

\directlua{
    local position_array = {}
    local function savepos_func()
      position_array[\csstring\#position_array+1] = {pdf.getpos()}
    end
    function getpos_and_reset()
      local oldpos = position_array
      position_array = {}
      return oldpos
    end
    local function my_post_lb_filter(head,groupcode)
      local HLIST = node.id("hlist") % node.id for a line of text in vertical list
      for n in node.traverse(head) do % For every subnode within node head
        if n.id==HLIST then % If its a line of text
          local savepos = node.new("whatsit","late_lua")
          savepos.data = savepos_func
          n.head = node.insert_after(n.head,node.tail(node.list),savepos)
        end
      end
      return head
    end
  luatexbase.add_to_callback('post_linebreak_filter', my_post_lb_filter, 'Play with luatex node library')
}


\begin{document}

\raggedright
\newbox\myoddvbox
\newbox\myoddvboxsplit
\global\setbox\myoddvbox=\vbox{{\hsize=2in \blindtext[1]\endgraf}}%
\setbox\myoddvboxsplit=\vsplit\myoddvbox to 4cm


\begin{textblock*}{0.5\linewidth}[0,0](2in,2in)
\begin{minipage}[t][1cm][t]{0.5\linewidth}%
  \box\myoddvboxsplit%
\end{minipage}
\end{textblock*}


\begin{textblock*}{0.5\linewidth}[0,0](5in,2in)
\begin{minipage}[t][1cm][t]{0.5\linewidth}%
  \box\myoddvbox%
\end{minipage}
\end{textblock*}


\null
\newpage

% replace following line with a iterate function to print all the positions from lastpage
\directlua{
  for _, pos in ipairs(getpos_and_reset()) do
    tex.sprint('(' .. tostring(pos[1]) .. ', ' .. tostring(pos[2]) .. ') ')
  end
}


\end{document}

A note about the use of local and global variables: As you have seen, I declared most variables to be local. This is generally a good idea to avoid polluting the global namespace. (It is also a tiny bit faster than using globals) Especially savepos_func is never used outside of the \directlua``block, so it is best declaredlocal. Similar formy_post_lb_filter`.

position_array can be local or global depending on when you need to access it. The current code requires you to reset the array on every page to get reliable entries, so getpos_and_reset is used as a global interface instead of making position_array directly accessible in an attempt to force people to reset the array. If you add another way of reliably resetting it before every shipout, you can safely make position_array global.