Issues with TeX sub-formula formatting

As generally acknowledged TeX offers superb formatting capabilities for math in an automated fashion. Nevertheless there are some areas with clear deficiencies that one either has to accept or manually improve on a case by case basis.

One of the biggies in that respect is TeX's handling of sub-formulas (i.e., material inside a brace group, e.g., and index, but also any group on top-level). TeX sets such sub-formulas always in natural width even if the whole formula is subject to severe stretching or shrinking otherwise. As an example, consider the following (not very sensible) example:

\documentclass{article}

\newlength\x

\begin{document}

\newcommand\formula{$a+\mathbf{b+c}+d+e+f = \sum_{i=1}^{n-x-y-z} x_{i+j}$}

\settowidth\x{\formula}

\hbox to \x{\formula}    % natural width

\hbox to \x{\formula}   % now shrink it a lot

\hbox to \x{\formula}   % now stretch a lot in the opposite direction

\end{document}


The formula here has a number of obvious sub-formulas in the subscripts and superscript, but for illustration I also added a sub-formula via \mathbf. Of course the usage of \mathbf in this way is wrong!!! (it should not be applied to several symbols), but I'm sure you would find this in documents. In any case just {b+c} would have had the same effect here. Now what do you think happens if we run this?

This:

As one can see the naturally "boxed" sub-formulas are very wrong the moment the rest of the formula is subject to stretching or shrinking.

Until recently no TeX engine successor addressed this issue. With LuaTeX opening up a lot of the internals of TeX I had some hope that this would be different. However, upon studying the manual my conclusion is that this area (sor far it least) has not been addressed (or considered).

As far as I can see the only way something could be done in LuaTeX about this issue would be to use the mlist_to_hlist callback. However, this would really mean replacing the full math typesetting algorithm, which of course could be a way to solve the problem but ... but what is needed is not that (as 99% of this algorithm is next to perfect) but to add support to not simply box sub-formulas at their natural width.

So long text ... here is the short question:

• Is this analysis correct, or did I overlook something?
• And in case anybody knows: are there plans to look into the sub-formula issue and provide support for it eventually? (it is not listed on the "to-dos" for math)

In case somebody wonders that I talk about "this sub-formula issue" as if it is something like a known thing ... it is, it was already raised way back in E-TeX: guidelines to future TeX extensions which at the time a some thorough (if not say heated) discussions.

-
This is the same behavior with \newcommand\formula{A \hbox{b c} d}... –  Paul Gaborit Jun 26 '12 at 22:50
@PolGab sure. and \hbox is what a sub-formula produces when TeX converts an mlist to an hlist. so from a conceptual perspective this is a natural thing to do as long as you think that sequential processing is the right kind of model (which is what I'm challenging (for 2 decades here)) –  Frank Mittelbach Jun 26 '12 at 23:19
what if you wanted to embolden a "word" in math, like \mathbf{Span}(x + y) ? (yes, i know it would be better to use \DeclareMathOperator, but we see this all the time.) you wouldn't want those letters to "fly apart". to me, this is the same as @RyanReich's \textbf{b c}. i'm afraid i think that the only "good" approach is for authors to learn to "do the right thing" (a goal of which i despair). –  barbara beeton Jun 27 '12 at 12:40
@barbarabeeton adjacent letters have no stretchable glue between them so there isn't a problem there and so you might hope that it worked just like a \begingroup \fam4 b+c\endgroup group which doesn't cause an internal box to be created so allows the operator spacing to stretch or shrink (but doesn't affect inter-letter spaces). –  David Carlisle Jun 27 '12 at 12:50
@barbarabeeton \mathbf{Span} as this is just a sequence of alphabet letters. The issue is with \mathbf{a+b} because this is a mix of a symbol which is not an alphabet char and letters and the + symbol is a binary and so has variable spacing to its left and right ... and those get frozen. –  Frank Mittelbach Jun 27 '12 at 20:04

Looks like the answer is as follows:

• This issue (which is present already in the original program of TeX) is not being solved or addressed by any TeX successor including LuaTeX.

It is true that LuaTeX offers to replace all of the math processing by proviate code but this is more along the lines "demolish the house and build a new one" and not really warranted. After all TeX's algorithm are really great in most respects. A pity, but then perhaps understandable as it would require to take the processing logic of the math formatting appart and reorganize it to improve only a small fraction of it.

-

There is a way to do this with unicode-math since bold letters in Unicode math are different letters of the same font rather than letters of a different font:

\documentclass{article}

\usepackage{unicode-math}

\setmathfont{XITS Math}

\newlength\x

\begin{document}

\newcommand\formula{$a+𝐛+𝐜+d+e+f = \sum_{i=1}^{n-x-y-z} x_{i+j}$}

\settowidth\x{\formula}

\hbox to \x{\formula}    % natural width

\hbox to \x{\formula}   % now shrink it a lot


I couldn't find a way to make this work with \mathbf, but it should be doable: Essentially \mathbf could be made into something like \begingroup \Umathcode a=𝐚 … \endgroup.
You are mistaken in what I'm trying to high-light: any of the current TeX-based engines treat sub-formulas (i.e., a straight brace group) as something that is typeset first (at natural width and inserted into the formula internally as an hbox) and such a sub-formula is then not taking part in any shrink or stretch. The \mathbf was just an example (of deliberate incorrect input) to show how additionally surprise sub-formulas could appear. No unicode-math is making the ^{n-x-y-z} shrink when the rest of the formula is squeezed. –  Frank Mittelbach Jun 27 '12 at 20:00
@Frank: +1 for your comment, although I should point out that there shouldn't be any shrinking in ^{n-x-y-z} anyway since in \scriptstyle there's no space around binary operations. Or am I mistaken? I think a better example would be a \left...\right construct. –  Hendrik Vogt Jun 27 '12 at 21:25