Missing } inserted and wrong conditional branch in align*

Question

\documentclass[a4paper]{article}
\usepackage{amsmath,amssymb,amsfonts}
\begin{document}
\begin{align*}
\eta_i\in{}&\ifmmode\mathcal C^\infty\else$\mathcal C^\infty$\fi
\end{align*}
\end{document}

Typing pdflatex in the command line and running this, I would expect it to work. align* is a math mode environment, so the \ifmmode should take the YES branch, and just produce \mathcal C^\infty. However, that is not the case. The result of the above is Missing } inserted, and if I add \show\mathcal right after the \else, the \show gets executed, meaning it's going into the NO branch as if it weren't in math mode. If I remove the &, however, everything works fine, and the result is:

Same goes with xelatex.

What is going on here? Why is it not in math mode after the &?

Answer 1

The align environment is defined in terms of the primitive function \halign.

When TeX is processing \halign and finds & it ignores following explicit spaces and expands tokens in order to see whether \omit or \span follows. Only after finding either of them or neither, it inserts the template for the current column.

The template for an even numbered column in align is basically

$\displaystyle#$\hfill

where # stands for the cell content, found by scanning to the next & or row ending.

What happens in your case is that \ifmmode is expanded and TeX is not in math mode at the time, so what remains is $\mathcal C^\infty$

Now TeX applies the template, resulting in

$\displaystyle$\mathcal C^\infty$$

which places \mathcal outside of math mode. Hence the error.

A naïvely defined macro

\newcommand{\Cinfty}{\ifmmode\mathcal{C}^\infty\else$\mathcal{C}^\infty$\fi}

would not solve the issue. The LaTeX kernel provides \ensuremath for the purpose:

\newcommand{\Cinfty}{\ensuremath{\mathcal{C}^\infty}}

But you should think twice (or more) before doing this as you have no advantage whatsoever in being able to type \Cinfty in text mode instead of the correctly marked up $\Cinfty$. Note that $\Cinfty$ and \Cinfty{} require the same number of keys.

Answer 2

I believe that you can't use the \ifmmode inside math, and this command is used to define something, the modified MWE is:

\documentclass[a4paper]{article}
\usepackage{mathtools}
\begin{document}
\newcommand{\calC}{\relax\ifmmode\mathcal{C}^{\infty}\else%
$\mathcal{C}^{\infty}$\fi}%


\begin{align*}
\eta_i\in{}&\calC
\end{align*}

\calC

\end{document}