Writing specific versions of an operator for display and inline modes

Question

I'm busy answering a question by constructing a new math operator that uses the sum symbol and some TikZ placed on top. My answer as it stands can be found here. It looks (I think) pretty good in display style, but too large when shown inline. Even if it automatically set the sum symbol into inline size, the placement of the TikZ would be off.

Is there a way to define different code for when the operator is used in display mathematics environments, and when it's inline? Alternatively, is there a way to detect when you are in display or inline mode?

From the aforementioned example, here is the code I constructed:

\documentclass{minimal}
\usepackage{amsmath}
\usepackage{tikz}

% Modify the coordinate (-0.3ex,0) to adjust alignment, and (0.1) to adjust the size.
\DeclareMathOperator*{\osum}{\ensuremath{
    \begin{tikzpicture}[baseline=(char.base), inner sep=0, outer sep=0]
        \draw (-0.3ex,0) circle (0.1); 
        \node (char) at (0,0) {$\displaystyle\sum$};  % Want to define a second symbol for inline...
    \end{tikzpicture}}%
}

\begin{document}

\[
G=\osum_a^b H
\]

And now in inline mode the equation $G=\osum_a^b H$ is obviously too big.

\end{document}

Answer 1

TeX features a special primitive for this very case. However, you will have to specify code for all four math styles:

\mathchoice{display}{text}{script}{scriptscript}

See TeX by Topic for more information.

Answer 2

Taking the definitions you make in the other question, here's a way: \mathchoice has four arguments, stating what's to be done in the various situations; \mathop states how the symbol should be considered with respect to spacing and ending with \displaylimits ensures the same behavior as \sum:

\documentclass{article}
\usepackage{tikz}
\newcommand{\osum}{ 
  \mathop{
    \mathchoice
      {\buildosum{\displaystyle}{0.1}}
      {\buildosum{\textstyle}{0.075}}
      {\buildosum{\scriptstyle}{0.075}}
      {\buildosum{\scriptscriptstyle}{0.075}}
  }\displaylimits 
}

\newcommand\buildosum[2]{%
  \begin{tikzpicture}[baseline=(char.base), inner sep=0, outer sep=0]
    \draw (-0.3ex,0) circle (#2);
    \node (char) at (0,0) {$#1\sum$};
  \end{tikzpicture}%
}

\begin{document}

\[\osum_a^b x\]

$\osum_a^b x$

\end{document}

If you need more parameters to adjust the circle's position in smaller styles, modify accordingly the definition. Using a two-step approach makes the code more easily maintainable.