Discrepancy between lualatex and xelatex regarding display math vertical spacing

Consider this

\documentclass[a4paper]{article}
\usepackage[left=0cm,right=12cm]{geometry}
\usepackage{unicode-math}
\setmathfont{Libertinus Math}
\usepackage[grid]{eso-pic}
\begin{document}
Before
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
After
\end{document}


At left, result from xelatex, at right from lualatex:

Testing was done with TL2018 (updated pretest). Do mentally the slight shift upwards of right image to perfectly align it with left. The point is that xetex output is more spread out vertically. In real life this means documents will differ easily in pagination.

It was pointed out in comment that this is already the case due to LuaTeX extending the hyphenation algorithm, whereas XeTeX sticks with the traditional one.

This testing came out of Problem with \underbrace and Libertinus Math

With default fonts, no discrepancy.

Sorry, edit: also discrepancy with commenting out \setmainfont{Libertinus Math} and keeping only unicode-math.

\documentclass[a4paper]{article}
\usepackage[left=0cm,right=12cm]{geometry}
\usepackage{unicode-math}
\usepackage[grid]{eso-pic}
\begin{document}
Before
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
$\underbrace{AAA}_{B}$
After
\end{document}


Editing question title to remove reference to Libertinus Math.

Using only fontspec but not unicode-math outputs are identical. Using fontspec and unicode-math (or only the latter which loads the former) the outputs differ even with no additional font setup.

Same issue when using for example \setmathfont{TeX Gyre Pagella Math}.

It is not an issue of unicode-math but an engine “problem” which is shown using this Plain TeX example:

\nopagenumbers
\ifdefined\directlua
\fi

% Load any Unicode math font (we don't need scriptscript here)
\font\mathfont="TeX Gyre Pagella Math:script=math" at 10pt
\textfont0=\mathfont
\font\mathfonts="TeX Gyre Pagella Math:script=math;+ssty=0" at 7pt
\scriptfont0=\mathfonts

% Define the relevant symbols
\UmathcodenumA="1D434
\UmathcodenumB="1D435
\def\underbrace#1{\mathop{\Umathaccent bottom 0 0 "23DF {#1}}}

$$\hbox{Before}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\hbox{After}$$
\bye


Left is LuaTeX, right is XeTeX:

• github.com/wspr/unicode-math/issues/453 – user4686 Mar 25 '18 at 13:24
• Math rendering in XeTeX is broken. This one has been fixed tex.stackexchange.com/questions/281549/…, this one hasn't tex.stackexchange.com/questions/374771/… (as far as I know). – Henri Menke Mar 25 '18 at 20:40
• @HenriMenke I bookmarked your tex.stackexchange.com/questions/281549/… long time ago ;-) good to know it has been fixed ! Thus, in the present case, your estimate is that LuaLaTeX+unicode-math gets the correct output and XeLaTeX the wrong one ? – user4686 Mar 25 '18 at 21:23
• It's not really a question of right and wrong here. The rendering of Unicode math fonts is approached entirely different by the two engines. It is rather a question of whether they are consistent. In XeTeX it has been observed that fontdimens are somehow dependent on the font size chosen, which is definitely an inconsistent behaviour. If I find time, I'll do some experiments on this case in Plain TeX (unicode-math does too much extra stuff, obfuscating what is actually going on). I highly doubt that the unicode-math package is at fault though. It seems much more like an engine problem. – Henri Menke Mar 25 '18 at 22:35
• I've edited your question with a Plain TeX example. I hope you don't mind. This probably means that you can close issue on GitHub for unicode-math. – Henri Menke Mar 26 '18 at 1:49

This is not an answer but more of a rhetorical follow-up question which is too long for a comment.

Both XeTeX and LuaTeX will emulate the traditional TeX behaviour of math typesetting in the absence of Unicode math fonts. We can exploit that to draw a side-by-side comparison of the output with and without a Unicode math font.

\nopagenumbers
\ifdefined\directlua
Lua%
\else
Xe%
\fi
\TeX

\hbox{
\vbox{\hsize=50pt
$$\hbox{Traditional}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\hbox{After}$$
}

\font\mathfont="Latin Modern Math:script=math" at 10pt
\textfont0=\mathfont
\font\mathfonts="Latin Modern Math:script=math;+ssty=0" at 7pt
\scriptfont0=\mathfonts

\UmathcodenumA="1D434
\UmathcodenumB="1D435
\def\underbrace#1{\mathop{\Umathaccent bottom 0 0 "23DF {#1}}}

\vbox{\hsize=50pt
$$\hbox{Unicode}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\underbrace{AAA}_{B}$$
$$\hbox{After}$$
}
}
\bye


Here we begin to run into some serious trouble because neither engine lines up with the traditional TeX output. This is partly due to the fact that the metrics are slightly different and the underbrace accent has a radically different shape. But at the same time this discrepancy makes it impossible to say which engine is “right”.

The only thing one could do at this point is ask the developers of both engines to review their code and see whether any wrong scalings creep in somewhere, but that seems a bit pointless to me because it is not really clear what we are looking for. Because XeTeX has kind of a history of broken math, I'd suspect the error with them, but that is purely a guess and might as well be a false accusation.