I probably should have posted this answer (Proper use of \mathchoice) here, but I will just refer you to it. It got adapted into the scalerel
package, where I introduce the syntax
\ThisStyle{...\SavedStyle...}
which can also be nested as
\ThisStyle{...\SavedStyle...\ThisStyle{...\SavedStyle...}...}
The invocation of \ThisStyle
saves the current math style, which can later be recalled via \SavedStyle
. A final noteworthy point is that this approach uses the TeX primitive \mathchoice
, which will not suffer the many compatibility issues that others have noted with the mathstyle
package.
Werner asks for an MWE here, so here is one, in which a math notation is introduced (a triple-stacked subscript) that works just fine in \textstyle
. But, as originally defined, changes in the math style cannot migrate into the stack, because it is formed inside a LaTeX box construct. Thus, to carry the mathstyle into the stack, the above-described syntax from the scalerel
package is introduced:
\documentclass{article}
\usepackage{scalerel}
\usepackage[usestackEOL]{stackengine}[2013-09-11]
\stackMath\setstackgap{S}{1pt}
\parindent 0in\parskip 1em
\begin{document}
A unique symbol in textstyle math\\
$A_{\Shortunderstack{a \\ b \\ c}}$
However, the stack doesn't see the scriptstyle\\
$\scriptstyle A_{\Shortunderstack{a \\ b \\ c}}$
Now it does:\\
\def\subterm{\ThisStyle{ A_{\Shortunderstack{%
\SavedStyle a \\ \SavedStyle b \\ \SavedStyle c}}}}
$\textstyle\subterm$
$\scriptstyle\subterm$
$\scriptscriptstyle\subterm$
In these cases, the subscript is one reduced from\\ the main text:\\
\def\subterm{A_{\ThisStyle{\Shortunderstack{%
\SavedStyle a \\ \SavedStyle b \\ \SavedStyle c}}}}
$\textstyle\subterm$
$\scriptstyle\subterm$
\end{document}

The actual construction of these macros in scalerel.sty
is straightforward. It uses \mathchoice
to define a switch at the invocation of \ThisStyle
called \m@switch
, defined as one of four unique letters, depending on the current math style. Then it proceeds with the argument of \ThisStyle
within the \mathchoice
. Upon coming across a \SavedStyle
within that argument, it uses the \m@switch
character to construct a macro name, by adding the switch character to the end of the string \@mstyle
. Those four variants \@mstyleD
, \@mstyleT
, \@mstyleS
, and \@mstyles
just regurgitate the math style which had been saved at the invocation of \ThisStyle
.
\def\@mstyleD{\displaystyle}
\def\@mstyleT{\textstyle}
\def\@mstyleS{\scriptstyle}
\def\@mstyles{\scriptscriptstyle}
%
\def\SavedStyle{\csname @mstyle\m@switch\endcsname}
%
\newcommand\ThisStyle[1]{%
\ifmmode%
\def\@mmode{T}\mathchoice%
{\edef\m@switch{D}#1}%
{\edef\m@switch{T}#1}%
{\edef\m@switch{S}#1}%
{\edef\m@switch{s}#1}%
\else%
\def\@mmode{F}%
\edef\m@switch{T}#1%
\fi%
}
\mathchoice
, but not to capture the current math style for possible later use.\over
tex.stackexchange.com/questions/42855/whats-behind-over/…