Using post_linebreak_filter in LuaTeX to set height and depth of boxes

Question

After reading this answer Automatically prevent extra line spacing because of math by David Carlisle, I experimented with the code to get grid typesetting. My approach: The height of a line is always (n - 0.3)\baselineskip and the depth (n - 0.7)\baselineskip with n ∈ ℕ. Then I delete the glue. This works fine for big inline math or other big obejects in horizontal mode, but it does not work with display math as you can see in the MWE.

So I have some questions:

1. What ist wrong here?
2. How can I influence the height and depth of a display math line in post_linebreak_filter or somewhere else?
3. How can I get the spacing right in this case? That means that the baseline of the display math should be on the grid and the lines after the display math should also obey the grid.

MWE:

\documentclass[12pt]{article}

\usepackage{showframe,color,unicode-math}

% Check grid typesetting
\def\grid{%
    \vtop to0pt{%
        \hsize=0pt \noindent \color{red}%
        X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\vss}}

\raggedbottom

\directlua{
function fixlines(h,c)
  local B=tex.baselineskip.width
  for n in node.traverse(h) do
  %
  % correct height and depth
   if (n.id==0 or n.id==11) then
    n.height = (math.ceil((n.height+.3*B)/B)-.3)*B
    n.depth  = (math.ceil((n.depth+.7*B)/B)-.7)*B
   end
%
% lineskip or baselineskip: kill glue
   if n.id==12 and (n.subtype==1 or n.subtype==2) then
    n.width=0 % kill glue
   end
  end
 return h
end
luatexbase.add_to_callback('post_linebreak_filter', fixlines, 'fix line spacing')
}

% test big objects
\def\𝟙{\begin{pmatrix}1&0&0\\0&1&0\\0&0&1\end{pmatrix}}

\usepackage{lipsum}

\begin{document}

\parskip0pt
\lineskip0pt
\lineskiplimit0pt
% displaymath spacing
\abovedisplayskip\baselineskip
\abovedisplayshortskip\baselineskip
\belowdisplayskip\baselineskip
\belowdisplayshortskip\baselineskip
    
\noindent \grid I like grid typesetting. Let us test some inline math. Let us
test some very big inline math with : $𝟙₃ = \𝟙$. The next line obeys the grid,
very good! 2 baselines were omitted above, one below.
    
What about \detokenize{$$}-math?
% Sorry for using $$, but I usually use Plain LuaTeX.
$$ a+b+c+d~\hbox{Why don't I have the same baseline as the red X?} $$
The grid is not obeyed any more. What is wrong here? How can I influence the
height and depth of a display math line and its spacing in \detokenize{post_linebreak_filter}?
    
\end{document}

Edit: I have added the following code:

\directlua{
local B=tex.baselineskip.width
function killdisplayskips(h)
 for n in node.traverse(h) do
  if n.id==12 and (n.subtype==4 or n.subtype==5 or n.subtype==6 or n.subtype==7) then
  n.width=0 % kill glue
  end
 end
 return h
end
luatexbase.add_to_callback("append_to_vlist_filter", killdisplayskips, "killdisplayskips")
}

The spacing is still wrong (but it changed).

Answer 1

After a sleepless night I found a solution. In the append_to_vlist_filter you can manipulate normal lines (n.subtype==1), equations (n.subtype==6, that means lines of display math) und equationnumbers (n.subtype==7, contents after \eqno or \leqno). Now every display math line has a corrected height and depth, too. This is grid typesetting!!!! :)

Complete MWE:

\documentclass[12pt]{article}

\usepackage{showframe,color,unicode-math}
\usepackage{lua-visual-debug}

% Check grid typesetting
\def\grid{%
    \vtop to0pt{%
        \hsize=0pt \noindent \color{red}%
        X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\\X\vss}}

\raggedbottom

\directlua{
function fixlines(h,c)
  local B=tex.baselineskip.width
  for n in node.traverse(h) do
  %
  % correct height and depth
   if n.id==0 and (n.subtype==1 or n.subtype==6 or n.subtype==7) then
    n.height = (math.ceil((n.height+.29*B)/B)-.3)*B
    n.depth  = (math.ceil((n.depth+.69*B)/B)-.7)*B
   end
%
% lineskip or baselineskip: kill glue
   if n.id==12 and (n.subtype==1 or n.subtype==2) then
    n.width=0 % kill glue
   end
  end
 return h
end
luatexbase.add_to_callback('append_to_vlist_filter', fixlines, 'fix line spacing')
}

% test big objects
\def\𝟙{\begin{pmatrix}1&0&0\\0&1&0\\0&0&1\end{pmatrix}}

\begin{document}

\parskip0pt
\lineskip0pt
\lineskiplimit0pt
\normallineskiplimit0pt
\jot0pt
% displaymath spacing
\abovedisplayskip\baselineskip
\abovedisplayshortskip\baselineskip
\belowdisplayskip\baselineskip
\belowdisplayshortskip\baselineskip
    
\noindent \grid I like grid typesetting. Let us test some inline math. Let us
test some very big inline math with : $𝟙₃ = \𝟙\displaystyle∫$. The next line obeys the grid, very good!
    
What about \detokenize{$$}-math?%\vrule depth.3\baselineskip width2pt
% Sorry for using $$, but I usually use Plain LuaTeX.
$$
X+A+a+b+c+d∬~\hbox{Now I have the same baseline as the red X!}
$$
The grid is obeyed after all diplay math lines. Their height and depth do not
matter any more!
    
\end{document}

Edit: The solution above suffers from gobbled baselineskip in some situations (see wipet's comment). I found out that you have to calculate the interline skips on your own when using append_to_vlist_filter, otherwise they are gobbled.

The calculation is taken from: Underline part of a word while preserving kerning

So replace the luacode in the MWE by:

\directlua{
function correctheight(h, loc)
  local B=tex.baselineskip.width
  for n in node.traverse(h) do
  % correct height and depth of lines (post_linebreak) and display math incl.
  % \(l)eqno
    if loc=='post_linebreak' or loc=='equation' or loc=='equation_number' then
    n.height = (math.ceil((n.height+.29*B)/B)-.3)*B
    n.depth  = (math.ceil((n.depth+.69*B)/B)-.7)*B
   end
  end
 return h
end
%
function makegrid(h, loc, prev, mirror)
 correctheight(h, loc)
% If append_to_vlist_filter is used then you have to calculate interline skips
% on your own, otherwise they are gobbled! 
 local new_prev = mirror and h.height or h.depth
  if prev > -65536000 then
   local lineglue = tex.baselineskip.width - prev - (mirror and h.depth or h.height)
   local skip
    if lineglue < tex.lineskiplimit then
     skip = node.new('glue', 1)
     node.setglue(skip, node.getglue(tex.lineskip))
    else
     skip = node.new('glue', 2)
     node.setglue(skip, node.getglue(tex.baselineskip))
     skip.width = lineglue
    end
   skip.next = h
   h = skip
  end
 return h, new_prev
end
luatexbase.add_to_callback('append_to_vlist_filter', makegrid, 'make grid')}

Answer 2

This is more of a comment rather than an answer. FWIW, grid typesetting with math works out of the box in ConTeXt:

\setuplayout[grid=yes]
\showgrid
\definemathmatrix[pmatrix][left={\left(\,}, right={\,\right)},simplecommand=PMATRIX]
\def\𝟙{\PMATRIX{1,0,0; 0,1,0; 0,0,1}}
\starttext
I like grid typesetting. Let us test some inline math. Let us
test some very big inline math with : $𝟙₃ = \𝟙\displaystyle∫$. The next line obeys the grid, very good!
    
What about \detokenize{$$}-math?
$$
X+A+a+b+c+d∬~\hbox{Now I have the same baseline as the red X!}
$$
The grid is obeyed after all diplay math lines. Their height and depth do not
matter any more!
    
\stoptext

which gives

You can also add \showboxes in the preamble to visualize the boxes (but it is a bit crowded with the grid lines)