Subtleties in placing superscripts: explaining what TeX is actually doing

Starting from this question I tried investigating what TeX is actually doing while typesetting different subscripts, ie, how it decides how much to shift the superscript up, in order to make it a superscript.

Also what font-related characteristics/dimensions are involved, and which one of these dimensions can be "safely" tweaked (ie, are specific to superscripts) and which one are, conversely, shared among the other math typesettings?

This is best done stepwise.

Step 0 - Notation and definitions

Any symbol fonts (ie font family 2 in TeX language) must have at least 22 parameters in three different "styles" in order for TeX to typeset a formula. The "styles" roughly can be grouped as "text" for "display style", "text style" and their cramped version (see TeXBook), "script" for "script style" (cramped and standard) and "scriptscript" (cramped and standard). The TeXBook describes extensively how to determine which style to use and how to determine the correct style switch from main formula to super/subscripts.

Anyway these parameters are called `fontdimenZ` where Z is the number (spanning from 1 to 22, as mentioned before) and will identify which parameter we will have to use.

Note: only family2 is used here and we're not concerned with family3 except in one case, which will be detailed later on explicitly

Now with the serious steps:

Step 1 - Determine the character-specific shift `u`

This is a quantity that is determined based on what the superscript is appended to. `u` is set to:

• 0 if we're appending the superscript to a single character,
• 0 if the supposed "math list" (roughly a grouped set of math symbols) is composed by a single ordinary symbol with no sub or superscripts, other than an accent atom (the list is never actually made)
• `h-SUPfontdimen18` when it's appended to another type of math list (practically any other case), where:

• `h` is the height of the box constructed for the list
• `SUPfontdimen18` is the `fontdimen18` for the style in which the superscript has to be set

This means that `a^2` will have an `u` = 0, while `{(a}^2` won't, since the math list contains more than one character.

Step 2 - Determine the style based shift `p`

This parameter is taken straight from the current style typeset and is:

• `fontdimen13` if the formula is typeset in display style

• `fontdimen15` if the formula is set in a cramped style

• `fontdimen14` if the formula is set in any other styles.

Step 3 - Determine the ex-height based shift `s`

Being `d` the depth of the superscript box, this is simply `d+(1/4)|fontdimen5|`.

Step 4 - Find the height without a subscript `k` (to compute anyway)

If no subscript is present, the amount that the superscript will be shifted up will be `k=max(u,p,s)`. Otherwise `k` will be computed the same way but will not be used yet.

Step 5 - Find the height with a subscript

This time it's needed a detour to fontfamily3, of which we will take `r=fontdimen8`, which represents the default rule thickness.

If the distance from the "tentative" upshifted character and the subscript is `≥4r` then the upshift is valid and we're done. Otherwise the subscript will get moved down in order to ensure the `4r` spacing is enforced, and the calculations proceed to step 5a.

Step 5a (not always done, see previous step) - Correction factor `q`

`q=(4/5)|fontdimen5|-k+d`. If this is greater than 0, then `k=k+q` and the subscript gets shifted up of the same quantity `q`. Everything is typeset.

Interconnections among parameters and other formula typesetting

Here I will briefly detail how the parameters are intertwined with the other formula typesetting calculations, in order to be able to use and adjust only the ones needed. Please mind that this disregards completely family 3 which must not be touched for this purpose, since its parameter is used very widely

• `fontdimen13`, `fontdimen14`, `fontdimen15`, and `fontdimen18` are specifically used in these calculations only and are in fact superscript parameters.

• `fontdimen5` represents the "x-height" and is thus widely used:

1. `(1/4) |(fontdimen 5)|` also takes part into determining the height of the radical rule in radicals (actually takes part in the minimum clearance that will be allowed between the radicand and the rule), and of course it determines the total Height (with capital H to disambiguate it from the normal height, ie, the "H" takes into account depth) of the radical symbol.
2. In the Accent atoms, it represents the maximum height of an accent atom, used to place a kern after an accent atom.
• Allow me some remarks. 1. A math list containing a single character is sometimes treated as it were simply a single, bare character (i.e., its “u-value” is set to zero), but sometimes not (i.e., the “u-value” is "SUPfontdimen18”, according to your notation. The details on when exactly this happens are found on The TeXbook, p. 290, lines 11–13. More precisely, what truly happens in those cases is that the wrapping list is removed and only its content, which is a single symbol, is added to the outer list, so my wording above actually misrepresented the real processing. – GuM May 24 '17 at 23:01
• @GustavoMezzetti in appendix G, rule 17 says that first the math list is typeset, then if its nucleus is a single symbol it is converted to a simple character box. Also page 290 states that the inner grouping is stripped only for ord atoms with no sup and subs, or ord with an acc. – Moriambar May 24 '17 at 23:04
• For example, try `\$|^{2}{|}^{2}{\mathord |}^{2}{\mathclose |}^{2}\$` and look at how the exponent are placed. This, by the way, explains why `\$\mathclose\rvert^{2}\mathclose{\rvert}^{2}\$` exhibits the issue, but `\$\mathclose\vert^{2}\mathclose{\vert}^{2}\$` does not. — I’m afraid I’ll have to defer the other remarks to tomorrow… – GuM May 24 '17 at 23:10
• Exact. So, neither in `\$|^{2}\$`, nor in `\${|}^{2}\$`, nor in `\${\mathord |}^{2}\$` a inner math list is generated, and Rule 17 receives an atom whose nucleus is “simply a symbol”; but in the case of `\${\mathclose |}^{2}\$` (or `\${\mathopen |}^{2}\$`, or `\${\mathbin |}^{2}\$`…) the inner list is not stripped off, and Rule 17 will process an atom having that list as its nucleus. But I repeat, it’s enough for tonight! – GuM May 24 '17 at 23:22
• @GustavoMezzetti ok corrected it – Moriambar May 24 '17 at 23:35