What are the rules for style change in formulas (Plain TeX)?

Question

Both numerator and denominator in the following two formulas are of the same size, namely \textstyle (from which we conclude, that: 1) style change does not affect the digit, which immediately follows; 2) style change does not propagate "under" \atop):

$$2\atop8$$
$$\displaystyle2\atop8$$

By comparing the following two formulas we show that rule 1) is violated:

$$\sum_{2\atop8}x$$
$$\sum_{\scriptstyle2\atop8}x$$

So, what are the exact rules for style change in formulas?

Answer 1

The font size for \displaystyle and \textstyle is the same. Therefore, a difference between the two formulas cannot be seen, if the material consists of digits only.

$$2\atop8$$
$$\displaystyle2\atop8$$

But if the sum with limits is used, then the position of the limits differ dependent on the style:

$$\sum_0^\infty$$
$$\textstyle \sum_0^\infty$$

This can be used as indicator for the \atop experiments:

$$             \sum_1^\infty \atop \sum_1^\infty$$
$$\displaystyle\sum_2^\infty \atop \sum_2^\infty$$

As can be seen, \displaystyle does not affect the whole formula. \atop splits the current formula in two parts, the upper and lower part. \displaystyle belongs to the upper part only. It needs to be repeated, when it should apply to the lower part also:

$$\displaystyle\sum_3^\infty \atop \displaystyle\sum_3^\infty$$

If the whole expression should be set in a different style, then \atop can be confined in a subformula via curly braces:

$$\textstyle {\sum_4^\infty \atop \sum_4^\infty}$$

Now \atop is set in \textstyle and it sets its upper and lower parts in \scriptstyle.

Answer 2

In fact \displaystyle, \textstyle, \scriptstyle and \scriptscriptstyle will be applied from the point on where they got invoked til the end of the formula or subformula.

In your first example \displaystyle gets selected by default, not \textstyle (where in this case the distinction does not make any difference as only digits get typeset, which are identical in both of the last mentioned styles.) Thus, you should not conclude rules from that example.

An example where you would actually see how the rules apply for the first case could be

$2\atop8$

\vskip 1em

$\displaystyle2\atop\displaystyle8$

\bye

And for the second one try something like

$$\sum_{2\atop8}x$$

$$\textstyle\sum_{2\atop8}x$$

$$\scriptstyle\sum_{2\atop8}x$$

\bye