How to capture the current math style?

Question

\mathchoice helps to distinguish between math styles:

\mathchoice
  {<something in \displaystyle>}
  {<something in \textstyle>}
  {<something in \scriptstyle>}
  {<something in \scriptscriptstyle>}

How would one use \mathchoice to capture the current math style? The following does not work:

\documentclass{article}
\makeatletter
\newcommand{\getmathstyle}{%
  \mathchoice
    {\global\def\curmathstyle{\displaystyle}}
    {\global\def\curmathstyle{\textstyle}}
    {\global\def\curmathstyle{\scriptstyle}}
    {\global\def\curmathstyle{\scriptscriptstyle}}
}
\makeatother
\begin{document}
\ttfamily
$\getmathstyle$ \meaning\curmathstyle\par
$\displaystyle\getmathstyle$ \meaning\curmathstyle\par
$\scriptstyle\getmathstyle$ \meaning\curmathstyle\par
$\scriptscriptstyle\getmathstyle$ \meaning\curmathstyle
\end{document}

Answer 1

I probably should have posted this answer (Proper use of \mathchoice) here, but I will just refer you to it. It got adapted into the scalerel package, where I introduce the syntax

\ThisStyle{...\SavedStyle...}

which can also be nested as

\ThisStyle{...\SavedStyle...\ThisStyle{...\SavedStyle...}...}

The invocation of \ThisStyle saves the current math style, which can later be recalled via \SavedStyle. A final noteworthy point is that this approach uses the TeX primitive \mathchoice, which will not suffer the many compatibility issues that others have noted with the mathstyle package.

---

Werner asks for an MWE here, so here is one, in which a math notation is introduced (a triple-stacked subscript) that works just fine in \textstyle. But, as originally defined, changes in the math style cannot migrate into the stack, because it is formed inside a LaTeX box construct. Thus, to carry the mathstyle into the stack, the above-described syntax from the scalerel package is introduced:

\documentclass{article}
\usepackage{scalerel}
\usepackage[usestackEOL]{stackengine}[2013-09-11]
\stackMath\setstackgap{S}{1pt}
\parindent 0in\parskip 1em
\begin{document}
A unique symbol in textstyle math\\
$A_{\Shortunderstack{a \\ b \\ c}}$

However, the stack doesn't see the scriptstyle\\
$\scriptstyle A_{\Shortunderstack{a \\ b \\ c}}$

Now it does:\\
\def\subterm{\ThisStyle{ A_{\Shortunderstack{%
  \SavedStyle a \\ \SavedStyle b \\ \SavedStyle c}}}}
$\textstyle\subterm$
$\scriptstyle\subterm$
$\scriptscriptstyle\subterm$

In these cases, the subscript is one reduced from\\ the main text:\\
\def\subterm{A_{\ThisStyle{\Shortunderstack{%
  \SavedStyle a \\ \SavedStyle b \\ \SavedStyle c}}}}
$\textstyle\subterm$
$\scriptstyle\subterm$
\end{document}

---

The actual construction of these macros in scalerel.sty is straightforward. It uses \mathchoice to define a switch at the invocation of \ThisStyle called \m@switch, defined as one of four unique letters, depending on the current math style. Then it proceeds with the argument of \ThisStyle within the \mathchoice. Upon coming across a \SavedStyle within that argument, it uses the \m@switch character to construct a macro name, by adding the switch character to the end of the string \@mstyle. Those four variants \@mstyleD, \@mstyleT, \@mstyleS, and \@mstyles just regurgitate the math style which had been saved at the invocation of \ThisStyle.

\def\@mstyleD{\displaystyle}
\def\@mstyleT{\textstyle}
\def\@mstyleS{\scriptstyle}
\def\@mstyles{\scriptscriptstyle}
%
\def\SavedStyle{\csname @mstyle\m@switch\endcsname}
%
\newcommand\ThisStyle[1]{%
  \ifmmode%
    \def\@mmode{T}\mathchoice%
      {\edef\m@switch{D}#1}%
      {\edef\m@switch{T}#1}%
      {\edef\m@switch{S}#1}%
      {\edef\m@switch{s}#1}%
  \else%
    \def\@mmode{F}%
    \edef\m@switch{T}#1%
  \fi%
}

Answer 2

The mathstyle package provides a means to tap into the current math style and redefines \mathchoice as a switch. As such, the following is a possible solution:

\documentclass{article}
\usepackage{mathstyle}% http://ctan.org/pkg/mathstyle
\makeatletter
\newcommand{\getmathstyle}{
  \global\edef\curmathstyle{\expandafter\@gobble\mathchoice{\@@displaystyle}{\@@textstyle}{\@@scriptstyle}{\@@scriptscriptstyle}}
}
\makeatother
\begin{document}
\ttfamily
\verb|\textstyle|: $xyz\getmathstyle$ \meaning\curmathstyle \par
\verb|\displaystyle|: $\displaystyle xyz\getmathstyle$ \meaning\curmathstyle \par
\verb|\scriptstyle|: $\scriptstyle xyz\getmathstyle$ \meaning\curmathstyle \par
\verb|\scriptscriptstyle|: $\scriptscriptstyle xyz\getmathstyle$ \meaning\curmathstyle
\end{document}

\mathpalette acts like a server for \mathchoice (see The mysteries of \mathpalette) and can be used to extract the math style in a slightly more elegant way:

%...
\usepackage{mathstyle}% http://ctan.org/pkg/mathstyle
\newcommand{\getmathstyle}{%
  \mathpalette{\global\let\curmathstyle}{\relax}%
}
%...

Answer 3

LuaTeX provides a \mathstyle primitive which takes the following values for different math styles:

• 0 = display
• 1 = crampeddisplay
• 2 = text
• 3 = crampedtext
• 4 = script
• 5 = crampedscript
• 6 = scriptscript
• 7 = crampedscriptscript

Here is an example usage:

\starttext
\startlines
$\displaystyle                  \sum_{k=0}^n (a_k + b_k): \mathstyle$
$\crampeddisplaystyle           \sum_{k=0}^n (a_k + b_k): \mathstyle$
$\textstyle                     \sum_{k=0}^n (a_k + b_k): \mathstyle$
$\crampedtextstyle              \sum_{k=0}^n (a_k + b_k): \mathstyle$
$\scriptstyle                   \sum_{k=0}^n (a_k + b_k): \mathstyle$
$\crampedscriptstyle            \sum_{k=0}^n (a_k + b_k): \mathstyle$
$\scriptscriptstyle             \sum_{k=0}^n (a_k + b_k): \mathstyle$
$\crampedscriptscriptstyle      \sum_{k=0}^n (a_k + b_k): \mathstyle$
\stoplines
\stoptext

which gives

You can do something based on the current mathstyle by using

\ifcase\mathstyle 
 ...
 \or
 ...
 \or
 ... % etc.
\fi

An interesting usage is

$\mathstyle_{\mathstyle_{\mathstyle}}$

which gives showing that cramped style is active in subscripts and subsubscripts.