The following MWE points out that, without fleqn class option, math internal mode is detected ("i" and "o") but, as soon as fleqn class option is enabled, this mode is not detected anymore ("i" and "i").
Do you see what's going on?
\documentclass
%[fleqn]
{article}
%
\newcommand{\test}{%
\relax%
\ifinner%
i%
\else%
o%
\fi%
}
%
\begin{document}
$\test$
%
\begin{equation}
\test
\end{equation}
\end{document}
\ifinner does not really test anything useful in math mode whether or not fleqn is used.
If you want to distinguish displaystyle then unfortunately you need to use \mathchoice.
The following produces t inline and d in display whether or not the [fleqn] option is used.
\documentclass
[fleqn]
{article}
%
\newcommand{\test}{%
\mathchoice
{\displaystyle d}%
{\textstyle t}%
{\scriptstyle s}%
{\scriptscriptstyle 2}%
}
%
\begin{document}
$\test$
%
\begin{equation}
\test
\end{equation}
\end{document}
\displaystyle vs non \displaystyle matters, isn't amsmath's \if@display reliable?
Commented
Mar 31, 2016 at 15:56
$$ or eqnarray or any other non-amsmath math construct then yes.
Commented
Mar 31, 2016 at 15:58
\ifinnerhas never been a good test for being in inline or display math.fleqnyou have forequationsomething like$\m@th\displaystyle \hskip\mathindent\bgroup.... Withoutfleqnyou have plain$$....fleqnyou will end up in$$...$$, withfleqnyou will end up in$...$. For more look atarticle.clsandfleqn.clo.\ifinneris a useful test in math mode.