How does one change the math style to one of the "primed" styles?

Question

Background
TeX typesets each portion of a mathematical expression in one of 8 styles:

1. display,
2. display',
3. text,
4. text',
5. script,
6. script',
7. scriptscript, or
8. scriptscript'.

Knuth calls them, D, D', T, T', S, S', SS, and SS', respectively.

TeX transitions between styles for subscripts, superscripts, and for the numerators and denominators of fractions (\above, \atop, \over and their \...withdelims variants).

Four of these style are well known via \displaystyle, \textstyle, \scriptstyle, and \scriptscriptstyle. The other four are less known (in fact, I don't think I've seen any reference to them outside of the TeXbook). They are almost the same as the corresponding nonprimed style except that superscripts are typeset a bit farther down.

One simple way to see the difference is to look at the output of (the plain TeX document)

$${\atop x^y}{\atop\textstyle x^y}$$
\bye

The reason these differ is that TeX starts the math display in style D. It typesets the numerator in style T and the denominator in style T'. The \textstyle causes TeX to typeset the denominator in style T instead.

Question How can one force TeX to go into one of the primed styles? As the example above illustrates, even when TeX is in one of the primed styles, it moves to the nonprime variant with \textstyle (similarly for the other three).

Motivation
This can be important when building vertical boxes in math mode. For example, see my answer to changing the shape of a fraction for a case where this is needed.

Follow up question

Is there a way to check if the current style is cramped? I thought about putting X^X in a \vcenter and comparing its height to the current style as determined by \mathchoice, but this doesn't work because \vcenter doesn't give you a box.

Answer 1

The mathtools package gives the \cramped command.

Answer 2

A totally different answer: after some toying around, I managed to produce this plain TeX macro:

\newdimen\savedrulethickness
\def\docramped#1#2%
  {\savedrulethickness = \fontdimen8\textfont3
   \hbox{$\mathsurround0pt 
          \fontdimen8\textfont3=0pt 
           #1\overline{#2}$}%
    \fontdimen8\textfont3 \savedrulethickness}

\def\cramped{\mathpalette\docramped}

$f^b \overline{f^b} \cramped{f^b}$

\bye

If math family 3 has three different fonts instead of just 3 copies of cmex10, this needs extending a bit, and there are the general pitfalls with \mathchoice, but it seems to work.

In comparison, here is how the definition of \cramped would look in Luatex:

\def\cramped#1%
  {{\ifcase\mathstyle
    \crampeddisplaystyle      \or \or % 0 -> 1
    \crampedtextstyle         \or \or % 2 -> 3
    \crampedscriptstyle       \or \or % 4 -> 5
    \crampedscriptscriptstyle \fi     % 6 -> 7
   #1}}

Edit: used \mathpalette, and added the Luatex version

Answer 3

\mathaccent's accentee is typeset in cramped style. So, if the math font has a character available that does not actually print anything, you could use that to force the style.

(luatex defines 4 new primitives for this purpose)

Answer 4

\radical0{...}

\radical is a TeX primitive used to define \sqrt as \def\sqrt{\radical"270370 }
(The TeXbook, p. 157).

The number identifies the square root symbol and zero is no error and no symbol.
This leaves the wanted side effect of typesetting the argument in the primed style (cramped).