\mathcal(O) and font size

Question

I use the following mystic TeX so as to display the Landau's notations for big and small O.

There is one partial solution to improve the scriptscriptstyle so as to not use a small o : see the secund answer below. The problemen with \smallOBis is the wrong spacing, contrary to \smallO. How can I do this ?

% Sources :
%     1) http://forum.mathematex.net/latex-f6/bonnes-commandes-de-base-t12278.html
%     2) http://tex.stackexchange.com/questions/30944/mathcalo-and-font-size

\documentclass{article}
    \usepackage{amsmath}
    \usepackage{amsfonts}
    \usepackage{amssymb}

    \usepackage{graphicx}

    \newcommand{\bigO}[1]{\ensuremath{\mathop{}\mathopen{}\mathcal{O}\mathopen{}\left(#1\right)}}

    \newcommand\smallO[1]{
        \mathchoice
            {% \displaystyle
                \ensuremath{\mathop{}\mathopen{}{\scriptstyle\mathcal{O}}\mathopen{}\left(#1\right)}
            }
            {% \textstyle
                \ensuremath{\mathop{}\mathopen{}{\scriptstyle\mathcal{O}}\mathopen{}\left(#1\right)}
            }
            {% \scriptstyle
                \ensuremath{\mathop{}\mathopen{}{\scriptscriptstyle\mathcal{O}}\mathopen{}\left(#1\right)}
            }
            {% \scriptscriptstyle  
                \ensuremath{\mathop{}\mathopen{}{o}\mathopen{}\left(#1\right)}
            }
    }

    \newcommand\smallOBis[1]{
        \mathchoice
        {% \displaystyle
            \scriptstyle\mathcal{O}\left(#1\right)
        }
        {% \textstyle
        \scriptstyle\mathcal{O}\left(#1\right)
        }
        {% \scriptstyle
        \scriptscriptstyle\mathcal{O}\left(#1\right)
        }
        {% \scriptscriptstyle
        \scalebox{0.8}{$\scriptscriptstyle\mathcal{O}$}\left(#1\right)
        }
    }


\begin{document}

\[
    \renewcommand{\arraystretch}{1.2}%
    \begin{array}{llll} 
        \textbf{Style}            & \verb!\bigO!
        & \verb!\smallO!                          & \verb!\smallOBis!
        \\ \hline
        \verb!\displaystyle!      & \displaystyle      \bigO{\frac{1}{n}}
        & \displaystyle      1 + \smallO{\frac{1}{n}} - 2 & \displaystyle      1 + \smallOBis{\frac{1}{n}} - 2
        \\
        \verb!\textstyle!         & \textstyle         \bigO{\frac{1}{n}}
        & \textstyle         1 + \smallO{\frac{1}{n}} - 2 & \textstyle         1 + \smallOBis{\frac{1}{n}} - 2
        \\
        \verb!\scriptstyle!       & \scriptstyle       \bigO{\frac{1}{n}}
        & \scriptstyle       1 + \smallO{\frac{1}{n}} - 2 & \scriptstyle       1 + \smallOBis{\frac{1}{n}} - 2
        \\
        \verb!\scriptscriptstyle! & \scriptscriptstyle \bigO{\frac{1}{n}}
        & \scriptscriptstyle 1 + \smallO{\frac{1}{n}} - 2 & \scriptscriptstyle 1 + \smallOBis{\frac{1}{n}} - 2
    \end{array}
\]

\end{document}

Answer 1

One problem here is the available font sizes at that small-a-size. Using a scaling factor supported by graphicx by means of \scalebox{<factor>}{<stuff>}:

\documentclass{article}
\usepackage{amsmath}% http://ctan.org/pkg/amsmath
%\usepackage{amsfonts}% http://ctan.org/pkg/amsfonts
%\usepackage{amssymb}% http://ctan.org/pkg/amssymb
\usepackage{graphicx}% http://ctan.org/pkg/graphicx

\newcommand{\bigO}[1]{\ensuremath{\mathop{}\mathopen{}\mathcal{O}\mathopen{}\left(#1\right)}}

\newcommand\smallO[1]{
    \mathchoice
    {% mode \displaystyle
      \scriptstyle\mathcal{O}\left(#1\right)
    }
    {% mode \textstyle
      \scriptstyle\mathcal{O}\left(#1\right)
    }
    {% mode \scriptstyle
      \scriptscriptstyle\mathcal{O}\left(#1\right)
    }
    {% mode \scriptscriptstyle
      \scalebox{0.8}{$\scriptscriptstyle\mathcal{O}$}\left(#1\right)
    }
}

\begin{document}

\[
  \renewcommand{\arraystretch}{1.2}%
  \begin{array}{lll}
    \textbf{Style} & \verb!\bigO! & \verb!\smallO! \\ \hline
    \verb!\displaystyle! & \displaystyle\bigO{n} & \displaystyle\smallO{n} \\
    \verb!\textstyle! & \textstyle\bigO{n} & \textstyle\smallO{n} \\
    \verb!\scriptstyle! & \scriptstyle\bigO{n} & \scriptstyle\smallO{n} \\
    \verb!\scriptscriptstyle! & \scriptscriptstyle\bigO{n} & \scriptscriptstyle\smallO{n}
  \end{array}
\]

\end{document}

The choice was made to scale a \scriptscriptstyle\mathcal{O} by 80%, since the shape of \mathcal{O} at larger font sizes looks different from that of \scriptscriptstyle. I'm sure that playing around with larger size and smaller scale factors would provide a suitable solution.

---

Edit: An update to the definition of \smallOBis, as defined in @projetmbc's updated question:

\documentclass{article}
\usepackage{amsmath}% http://ctan.org/pkg/amsmath
\usepackage{graphicx}% http://ctan.org/pkg/graphicx

\newcommand{\bigO}[1]{\ensuremath{\mathop{}\mathopen{}\mathcal{O}\mathopen{}\left(#1\right)}}

\newcommand\smallO[1]{
  \mathchoice
  {% \displaystyle
    \mathop{}\mathopen{}{\scriptstyle\mathcal{O}}\mathopen{}\left(#1\right)
  }
  {% \textstyle
    \mathop{}\mathopen{}{\scriptstyle\mathcal{O}}\mathopen{}\left(#1\right)
  }
  {% \scriptstyle
    \mathop{}\mathopen{}{\scriptscriptstyle\mathcal{O}}\mathopen{}\left(#1\right)
  }
  {% \scriptscriptstyle  
    \mathop{}\mathopen{}{o}\mathopen{}\left(#1\right)
  }
}

\newcommand\smallOBis[1]{
  \mathchoice
  {% \displaystyle
    \operatorname{\scriptstyle\mathcal{O}}\!\left(#1\right)
  }
  {% \textstyle
      \operatorname{\scriptstyle\mathcal{O}}\!\left(#1\right)
  }
  {% \scriptstyle
      \operatorname{\scriptscriptstyle\mathcal{O}}\left(#1\right)
  }
  {% \scriptscriptstyle
      \operatorname{\scalebox{0.8}{$\scriptscriptstyle\mathcal{O}$}}\left(#1\right)
  }
}

\begin{document}

\[
  \renewcommand{\arraystretch}{1.2}%
  \begin{array}{llll} 
    \textbf{Style}            & \verb!\bigO!
    & \verb!\smallO!                          & \verb!\smallOBis!
    \\ \hline
    \verb!\displaystyle!      & \displaystyle      \bigO{\frac{1}{n}}
    & \displaystyle      1 + \smallO{\frac{1}{n}} - 2 & \displaystyle      1 + \smallOBis{\frac{1}{n}} - 2
    \\
    \verb!\textstyle!         & \textstyle         \bigO{\frac{1}{n}}
    & \textstyle         1 + \smallO{\frac{1}{n}} - 2 & \textstyle         1 + \smallOBis{\frac{1}{n}} - 2
    \\
    \verb!\scriptstyle!       & \scriptstyle       \bigO{\frac{1}{n}}
    & \scriptstyle       1 + \smallO{\frac{1}{n}} - 2 & \scriptstyle       1 + \smallOBis{\frac{1}{n}} - 2
    \\
    \verb!\scriptscriptstyle! & \scriptscriptstyle \bigO{\frac{1}{n}}
    & \scriptscriptstyle 1 + \smallO{\frac{1}{n}} - 2 & \scriptscriptstyle 1 + \smallOBis{\frac{1}{n}} - 2
  \end{array}
\]
\end{document}

Two changes were made to \smallOBis:

• Declared the resized \mathcal{O} as an operator via \operatorname{..}, therefore correcting the spacing with other relational/binary operators on the left side; and
• Corrected for spacing between new operator and brackets on the right side with \!.

Answer 2

This is a case that shows why Landau chose O and o and not other fancy alphabets: an uppercase O is quite easily distinguishable from a lowercase o.

So it would suffice

\newcommand{\bigO}[1]{O{\left(#1\right)}}
\newcommand{\smallO}[1]{o{\left(#1\right)}}

or, if one feels the necessity of spacing the "O" or "o" from what precedes it (a factor, for instance), use

\newcommand{\bigO}[1]{\mathop{}\!O{\left(#1\right)}}
\newcommand{\smallO}[1]{\mathop{}\!o{\left(#1\right)}}

No \ensuremath (it doesn't hurt, but it's better to segregate math), no other complicated maneuvers with \mathop and \mathopen.

In case \mathcal{O} is really needed, I wouldn't use it for the "small o", particularly if these objects must be used in subscripts or superscripts. Nevertheless, here is a simpler definition (requires graphicx):

\newcommand{\bigO}[1]{\mathcal{O}{\left(#1\right)}}
\newcommand\smallO[1]{
  \mathchoice
    {{\scriptstyle\mathcal{O}}}% \displaystyle
    {{\scriptstyle\mathcal{O}}}% \textstyle
    {{\scriptscriptstyle\mathcal{O}}}% \scriptstyle
    {\scalebox{.7}{$\scriptscriptstyle\mathcal{O}$}}%\scriptscriptstyle
  {\left(#1\right)}}

possibly with \mathop{}\! as before.

Here is a showcase; above the first way, below the "script O"