\cramped in fraction denominator doesn’t line up with regular fraction

Question

In a display, the denominator of a fraction is set in “cramped” \textstyle, which can be emulated with \cramped[\textstyle]{...} from the mathtools package. But sometimes the emulation is perfect, while sometimes it shifts the denominator down. Why?

\documentclass{article}
\usepackage{color}
\usepackage{mathtools}

\newcommand\drawbaseline[1]{%
  \ifvmode\leavevmode\fi
  \rlap{{\color{red}\vrule width #1 height 0.1pt }}%
}

\begin{document}

The denominators line up in this example:
\[
\frac{k^{2^n}}{\drawbaseline{45pt}k^{2^n}}\quad
\frac{k^{2^n}}{\cramped[\textstyle]{k^{2^n}}}
\]
The denominators don't line up in this example:
\[
\frac{\int_a^b k^2 g(x) f(x)\,dx}%
  {\drawbaseline{145pt}\!% <- cancels \, between \hbox{} and \mathop
   \int_a^b k^2 f(x)\,dx}\quad
\frac{\int_a^b k^2 g(x) f(x)\,dx}%
  {\cramped[\textstyle]{\int_a^b k^2 f(x)\,dx}}
\]
Why?

\end{document}

Added tex-core tag because I suspect this has something to do with the primitive \over.

---

Added clarifications

The problem became clearer after @wipet’s answer and my digging through tex.web and mathtools.dtx. In the second display of my MWE above, the latter denominator has an excessive height of exactly 1.25\fontdimen8\textfont3, which comes from \radical0 {...} (which in turn is what \cramped uses to emulate cramped styles).

Now, \cramped does try to correct this excessive height (and is ususally successful), but any later unboxing will ruin such correction because the height information will be lost. This “unboxing” behaviour is, by mathtools, referred to as

a most unfortunate TeX “feature.” (The faulty reboxing procedure.)

My original question “sometimes the emulation is perfect, while sometimes it shifts the denominator down” was only partially answered by @wipet. Why does TeX’s rebox subroutine seemingly behave differently?

Since @wipet also proposed a solution (i.e., replacing \box\z@ with \hbox{\box\z@}), I wondered if this was the best solution: Note that mathtools’ existing code seems to suggest {}\box\z@, which is a construction used in \mathXlap and \crampedXlap.

Answer 1

It seems to be a bug in mathtools.sty. The cramped style cannot be given as "native" in classical TeX but it must be emulated by \radical primitive with empty radical symbol. But the constructed box is

\hbox{empty root symbol\vbox{invisible rule (with zero height)
                             \kern normalrulewidth+something
                             \hbox{$given math material$}}}

so, the resulting box is higher by normalrulewidth+something. What exactly means normalrulewith+something is described in TeXbook, Appendix G, step 11.

The mathtools.sty tries to correct this box height by the line 1449 and 1450 of its code (version 2020/03/24 v1.24):

    \advance\dimen@-\ht\z@ \ht\z@=-\dimen@
    \box\z@

The problem is that the resulting \box\z@ is reboxed later, so this correction is lost. You can correct this, if you create your own copy of mathtools.sty in your working directory and change the lines 1449 and 1450 to:

   \advance\dimen@-\ht\z@ \ht\z@=-\dimen@
   \hbox{\box\z@}%

Your first example $k^{2^n}$ is created by somewhat different process and the radical+corrected box is not reboxed later, so the result is correct.

The question "why the box is reboxed sometimes" is answered in Appendix G of the TeXook, step 15a: if w(numerator)<w(denominator) rebox numerator to the w(denominator), if w(numerator)>w(denominator) rebox denominator to the w(nominator). I add note missing in 15a: \hss at both sides on box contents is added during reboxing.

Answer 2

@RuixiZhang If you believe there is a bug in The TeXbook and/or tex.web (i.e., they don't match), please send me (karl at freefriends dot org) a minimal plain TeX example and description of the discrepancy. Thanks.