Math operator resizing in autosized brackets

Question

I want to define a new command to format \mathbb{E} in the same way as \sum - same size in display/inline formulae, and same treatment of sub/superscripts. The following seems to work...

\newcommand*{\bbe}{%
   \ifinner
      \mathbb{E}%
   \else
      \mathop{\displaystyle\vcenter{\hbox{\huge$\mathbb{E}$}}}%
   \fi
}

...except that in display formulae, when inside automatically sized brackets (e.g. \left( \bbe_x \right)) it reverts to treating it as if it were in an inline formula (so smaller size, subscripts to the right not beneath).

I'm not sure what's causing this issue; any advice would be greatly appreciated!

Answer 1

The \ifinner test is not safe for this application. Indeed the conditional is true whenever TeX opens a new math list when it's already in math mode.

The following Plain TeX code shows this:

\def\x{\ifinner\message{inner}\else\message{outer}\fi}
$${\x}$$

will print inner in the log file. The same happens inside a \left-\right pair.

What you need is \mathchoice, which depends only on the current math style, which doesn't change in math sublists.

\newcommand*{\bbe}{%
  \mathop{
    \mathchoice{\vcenter{\hbox{\huge$\mathbb{E}$}}}
               {\kern0pt\mathbb{E}}
               {\kern0pt\mathbb{E}}
               {\kern0pt\mathbb{E}}
  }\displaylimits
}

Probably \larger from the relsize package is better for display style:

\usepackage{relsize}
\newcommand*{\bbe}{%
  \mathop{
    \mathchoice{\vcenter{\hbox{\larger[4]$\mathbb{E}$}}}
               {\kern0pt\mathbb{E}}
               {\kern0pt\mathbb{E}}
               {\kern0pt\mathbb{E}}
  }\displaylimits
}

Answer 2

\ifinner is the wrong sort of test here, you need \mathchoice

\documentclass{article}
\usepackage{amssymb}
\def\xbbe#1#2{\hbox{#1$#2\mathbb{E}$}}
\def\bbe{\mathop{%
\mathchoice
{\xbbe\huge\displaystyle}%
{\xbbe{}\displaystyle}%
{\xbbe{}\scriptstyle}%
{\xbbe{}\scriptscriptstyle}%
}}

\begin{document}

$a \bbe_0^1 $

\[ \bbe_0^1 \qquad \frac{\bbe_0^1}{\bbe_0^1} \]

\end{document}