Proper use of \mathchoice

Question

I am trying to learn how to use \mathchoice.

Before the solution was posted for Non-invasive replacement for \fbox?, I attempted to use mathchoice to adjust the size of the parameter before measuring its height, depth and width. To simply things, I first defined:

\newcommand{\SetHeightDepthWidth}[1]{%          
    \settoheight{\fillbox@height}{#1}%   
    \settodepth{\fillbox@depth}{#1}%     
    \settowidth{\fillbox@width}{#1}%     
}%

Then adjusted the beginning of the \fillbox macro as:

\newcommand{\fillbox}[1]{%
  \ifmmode%                                             
    \mathchoice%                                        
        {\SetHeightDepthWidth{$\displaystyle#1$}}%      
        {\SetHeightDepthWidth{$\textstyle#1$}}%         
        {\SetHeightDepthWidth{$\scriptstyle#1$}}%       
        {\SetHeightDepthWidth{$\scriptscriptstyle#1$}}% 
  \else%                                                
    \SetHeightDepthWidth{#1}%                           
  \fi%                                                  

Before these changes, the output was:

But with the above changes I get:

So, what happened? To me it appears to be some problem related to the grouping due to the extra brace groups required to pass in the four parameters to \mathchoice, but don't know how to fix it. Does \mathchoice behave differently than many of the other types of conditional macros such as \iftoggle{condition}{code if true}{code if false} from the etoolbox package?

Code:

The new code is marked and the old code is left commented to simplify comparison.

\documentclass{article}
\usepackage{tikz}

\makeatletter
  \newlength\fillbox@height%
  \newlength\fillbox@depth%
  \newlength\fillbox@width%
  \newcommand{\SetHeightDepthWidth}[1]{%   new
      \settoheight{\fillbox@height}{#1}%   new
      \settodepth{\fillbox@depth}{#1}%     new
      \settowidth{\fillbox@width}{#1}%     new
  }%

  \newcommand{\fillbox}[1]{%
  \ifmmode%                                             new
    \mathchoice%                                        new
        {\SetHeightDepthWidth{$\displaystyle#1$}}%      new
        {\SetHeightDepthWidth{$\textstyle#1$}}%         new
        {\SetHeightDepthWidth{$\scriptstyle#1$}}%       new
        {\SetHeightDepthWidth{$\scriptscriptstyle#1$}}% new
  \else%                                                new
    \SetHeightDepthWidth{#1}%                           new
  \fi%                                                  new
  %\ifmmode%
  %  \settoheight{\fillbox@height}{$#1$}%
  %  \settodepth{\fillbox@depth}{$#1$}%
  %  \settowidth{\fillbox@width}{$#1$}%
  %\else%
  %  \settoheight{\fillbox@height}{#1}%
  %  \settodepth{\fillbox@depth}{#1}%
  %  \settowidth{\fillbox@width}{#1}%
  %\fi%
  \raisebox{-\fillbox@depth}{%
    \begin{tikzpicture}[scale=1]
    \pgfsetfillcolor{yellow!90!red}
    \pgfsetfillopacity{0.5}
    \pgfsetrectcap
    \fill (0,-\fillbox@depth) rectangle (\fillbox@width,\fillbox@height);
    \pgfsetfillcolor{yellow!50!black}
    \pgfsetfillopacity{1}
    \fill (0,-\fillbox@depth)
          rectangle (\fillbox@width,-\fillbox@depth+.1pt);
    \fill (0,\fillbox@height)
          rectangle (\fillbox@width,\[email protected]);
    \fill (0,-\fillbox@depth)
          rectangle (.1pt,\fillbox@height);
    \fill (\fillbox@width,-\fillbox@depth)
          rectangle (\[email protected],\fillbox@height);
    \end{tikzpicture}%
    \kern-\fillbox@width%
  }%
  #1%
}
\makeatother

\begin{document}

\vskip1em
$\fillbox{p}_{\fillbox{x}}
  \mathrel{\stackrel{\fillbox{_{~+}}}{\fillbox{\leftarrow}}}
  \fillbox{(}\fillbox{\frac{\fillbox{1}}{\fillbox{2}}}\fillbox{\cdot}
  \fillbox{a}_{\fillbox{x}}\fillbox{\cdot}%
  \fillbox{\Delta}\fillbox{t}^{\fillbox{2}}\fillbox{)} \fillbox{+}
  \fillbox{(}\fillbox{v}_{\fillbox{x}}\fillbox{\cdot}%
  \fillbox{\Delta}\fillbox{t}\fillbox{)}$\fillbox{;}\par
$\fillbox{v}_{\fillbox{x}}
  \mathrel{\stackrel{\fillbox{_{~+}}}{\fillbox{\leftarrow}}}
  \fillbox{a}_{\fillbox{x}}\fillbox{\cdot}%
  \fillbox{\Delta}\fillbox{t}$\fillbox{;}\par
%    
%            This section is not relevant to the problem here.
%\vskip1em
%$p_x \mathrel{\stackrel{_{~+}}{\leftarrow}}
%  (\frac{1}{2}\cdot a_x\cdot\Delta t^2) + (v_x\cdot\Delta t)$;\par
%$v_x \mathrel{\stackrel{_{~+}}{\leftarrow}} a_x\cdot\Delta t$;\par
%
\end{document}

Answer 1

Everything I know about \mathchoice I learned from David Carlisle's accepted answer above. He explained it wonderfully. So this answer is not an attempt to replace that answer, but to amplify it. His key phrase that triggered my light bulb was "\mathchoice{a}{b}{c}{d} is indistinguishable from within tex from \hbox{a}\hbox{b}\hbox{c}\hbox{d}" which made me understand that the macro to be typeset had to go along with the \mathchoice invocation.

Because his answer is so beautifully concise, it took a while to dawn on me how to use \mathchoice in more complicated ways. To that end, I provide the command \MS that obtains/saves the current mathstyle and executes its argument. The command \SavedMathStyle can be used in the argument to recall the math style at time of invocation.

The command \MS can be used in a nested (recursive) manner, depending on whether one wishes to reset the saved math style at some midpoint in the argument to the outer invocation. The key feature of this solution is that, within the argument, the macro \SavedMathStyle will recall the math style associated with that level of nesting invocation. To make the point even more emphatic, I create an example wherein I place material inside a box in math mode, where clearly the original math style would otherwise be lost.

\documentclass{article}
\usepackage{graphicx}
\makeatletter
\def\@mstyleD{\displaystyle}
\def\@mstyleT{\textstyle}
\def\@mstyleS{\scriptstyle}
\def\@mstyles{\scriptscriptstyle}
%
\def\SavedMathStyle{\csname @mstyle\m@switch\endcsname}
%
\newcommand\MS[1]{\mathchoice%
  {\edef\m@switch{D}#1}%
  {\edef\m@switch{T}#1}%
  {\edef\m@switch{S}#1}%
  {\edef\m@switch{s}#1}%
}
%
% A SAMPLE COMMAND
\newcommand\mycmd[2]{\SavedMathStyle#2+\fbox{$\SavedMathStyle\frac{#1}{1+#2}$}}
%
\makeatother
\begin{document}

Inline \(\MS{\mycmd{x}{z}}\) test

displaystyle test\[\MS{\mycmd{x}{z}}\]

Recursive test (resetting mathstyle):\[\MS{\mycmd{\MS{\mycmd{x}{z}}}{y}}\]

Remaining examples keep original mathstyle throughout recursion.

Recursive test:\[\MS{\mycmd{\mycmd{x}{z}}{y}}\]

scriptsize test:\[x_{\MS{\mycmd{\mycmd{x}{z}}{y}}}\]

scriptscriptsize test:\[x_{y_{\MS{\mycmd{\mycmd{x}{z}}{y}}}}\]

Inline \(\MS{\mycmd{\mycmd{x}{z}}{y}}\) test

\end{document}

As a follow up, I would like to address Peter Grill's concerns. First, I will say that I have (as of today) adopted this approach in V1.5 of the scalerel package, because using the mathstyle package was causing incompatibilities ((Nonfatal but Symptomatic) Conflict of amsmath and mathstyle Packages). In the scalerel implementation, the invoking command has been named \ThisStyle and the recall command is named \SavedStyle (so I'm sorry for the nomenclature change through the course of this question). The following example

\documentclass{article}
\usepackage{scalerel}
\begin{document}

\[
\ThisStyle{
  \frac{x}{y}\scriptstyle
  \ThisStyle{\left(
    \frac{y}{z}\SavedStyle\frac{y}{z}
  \right)}
  \SavedStyle\frac{p}{q}
}
\]

\end{document}

shows this result

What we see is that the outer invocation of \ThisStyle was made in \displaystyle and the inner invocation in \scriptstyle. Within the inner invocation, \SavedStyle deploys \scriptstyle and in the outer invocation, it deploys \displaystyle, even after the nested invocation in a different style.

Answer 2

\mathchoice acts at the end of the math list when TeX is making a horizontal list by putting the math boxes in the right places. So the "choice" happens way after you are attempting to use the lengths. (They are being set in a group and left as zero, but you can't simply make a global assignment as the timing is all wrong.)

Look at any uses of \mathchoice (or its wrapper \mathpalette) in plain or latex.ltx. The usual idiom is that you need to put the entire command including all its arguments so, using fbox for simplicity:

\def\fa#1{%
\mathchoice
  {\fbox{$\displaystyle#1$}}%
  {\fbox{$\textstyle#1$}}%
  {\fbox{$\scriptstyle#1$}}%
  {\fbox{$\scriptscriptstyle#1$}}}

would work, although usually you simplify things by defining a helper macro that takes the math style as an argument, for example in latex.ltx \smash in math mode uses this form with \mathsm@sh being a version of \smash that takes (eg) \scriptstyle as an additional argument. So something like

\def\xfb#1#2{\fbox{$#1#2$}}

\def\fb{\mathpalette\xfb}

Or you can let the AMS take care of you

\usepackage{amsmath}

\def\fc#1{\text{\fbox{$#1$}}}