Adaptively sized delimiters that don't go beyond the size of \bigl and \bigr

Question

I would like adaptively sized delimiters, but only up to \bigl/\bigr-size. That is, there is variation only between ordinary/normal size and one size higher.

\documentclass{article}
\usepackage{amsmath}

\newcommand*{\abs}[1]{\left\lvert#1\right\rvert}


\begin{document}

\(\abs{M} + \abs{M^2} + \abs{\abs{M^2}^2}\)
  % the outer delimiters on the right are of the size of \Bigl and \Bigr
  % (i.e. the next larger size after that of \bigl and \bigr)

\(\abs{M} + \abs{M^2} + \bigl\lvert \abs{M^2}^2 \bigr\rvert\)

\end{document}

How can I achieve this? For the above code, this would amount to defining \abs in a way that makes the two lines look the same.

(Here the default behavior of \left and \right (i.e. scaling to \bigl/\bigr) looks better, but I'm trying to keep the example short; I'd have to load a lot of packages to demonstrate my personal need. Also, a simple reason is that I need to embed certain formulas in paragraph text.)

Answer 1

The following example measures the formula inside \abs with \left/\right and with \bigl/\bigr. If the version \left/\right is larger, then the version with \bigl/\bigr is used.

\documentclass{article}
\usepackage{amsmath}
\usepackage{mleftright}

\makeatletter
\newcommand*{\abs}[1]{%
  \mathpalette\@abs{#1}%
}
% Version with the horizontal spacing of `\left` and `\right` in all cases:
% \newcommand*{\abs}[1]{%
%   \mathinner{\mathpalette\@abs{#1}}%
% }
\newcommand*{\@abs}[2]{%
  \sbox0{$\m@th#1\mleft\lvert#2\mright\rvert$}%
  \sbox2{$\m@th#1\bigl\lvert#2\bigr\rvert$}%   
  \ifdim\wd0>\wd2 %
    \bigl\lvert#2\bigr\rvert
  \else
    \ifdim\ht0>\ht2 %
      \bigl\lvert#2\bigr\rvert
    \else
      \ifdim\dp0>\dp2 %
        \bigl\lvert#2\bigr\rvert
      \else
        \mleft\lvert#2\mright\rvert
      \fi
    \fi
  \fi
}
\makeatother

\begin{document}

\(\abs{M} + \abs{M^2} + \abs{\abs{M^2}^2}\)

\(\abs{M} + \abs{M^2} + \bigl\lvert \abs{M^2}^2 \bigr\rvert\)

\end{document}

Remarks:

• Instead of \left/\right macros \mleft/\mright of package mleftright are used to avoid the additional spacing of \left/\right.

• The method is not "bulletproof". It makes the following assumption. If the delimiters, picked by \mleft and \mright, are larger than the version with \bigl and \bigr, then this case can only be detected, if at least either the width, height or depth increases. For the case that the width remains the same, then the larger delimiters need to be slightly larger than the formula, otherwise the formula would hide the height/depth of the delimiters. I haven't analyzed all resizeable delimiters, but I think, usually the width of the delimiter will become larger that is easily detected.

Second approach

This approach implements the algorithm of minimal delimiter size according to "The TeXbook" by D. E. Knuth, "Appendix G: Generating Boxes from Formulas", item "19.":

If the math list begins and ends with boundary items, compute the maximum height h and depth d of the boxes in the translation of the math list [...]. Let a = σ₂₂ be the axis height, and let δ = max(h-a, d+a) be the amount by which the formula extends away from the axis. Replace the boundary items by delimiters whose height plus depth is at least max(⎣δ/500⎦f, 2δ-l), where f is the \delimiterfactor and l is the \delimitershortfall. [...]

It can deal correctly with an expression like M^{2^{2^2}}_{f_{f_f}}, which trips up the first approach by violating its assumption about delimiter dimensions.

Example file:

\documentclass{article}
\usepackage{amsmath}
\usepackage{mleftright}

\makeatletter
\newcommand*{\abs}[1]{%
  \mathpalette\@abs{#1}%
}
% Version with the horizontal spacing of `\left` and `\right` in all cases:
% \newcommand*{\abs}[1]{%
%   \mathinner{\mathpalette\@abs{#1}}%
% }
\newcommand*{\@abs}[2]{%
  % a := math axis height -> \dimen0
  \sbox0{$#1\vcenter{}$}%
  \dimen0=\ht0 %
  % formula without delimiters -> \box0
  % h := height of formula -> \ht0
  % d := depth of formula -> \dp0
  \sbox0{$#1#2$}%
  % delta := max(h-a, d+a) -> \dimen2
  \dimen2=\dimexpr\ht0-\dimen0\relax
  \dimen4=\dimexpr\dp0+\dimen0\relax
  \ifdim\dimen4>\dimen2 %
    \dimen2=\dimen4 %
  \fi
  % delimiter's total height >= max(floor(delta/500)*f, 2*delta - l)
  %   -> \dimen4
  % f := \delimiterfactor
  % l := \delimitershortfall
  \dimen4 = \dimen2 %
  \divide\dimen4 by 500 %
  \dimen4=\delimiterfactor\dimen4 %
  \dimen6=\dimexpr 2\dimen2 - \delimitershortfall\relax
  \ifdim\dimen6>\dimen4 %
    \dimen4=\dimen6 %
  \fi
  % formula with \bigl/\bigr
  \sbox0{$#1\bigl\lvert\bigr\rvert$}%
  % comparison: Use \mleft/\mright, if \bigl/\bigr is larger than needed
  \ifdim\dimexpr\ht0+\dp0\relax>\dimen4 %
    % Case 1: \bigl/\bigr is larger than the minimum needed height.
    %   This means that either (\lvert,\rvert) or (\bigl\lvert,\bigr\rvert)
    %   provide the smallest delimiter size that meet the minimum height requirement.
    %   Consequently we can have \mleft/\mright choose (they won't pick anything
    %   larger than \bigl/\bigr).
    \mleft\lvert#2\mright\rvert
  \else
    % Case 2: \bigl/\bigr is exactly right or smaller than the required minimum height
    %   => use \bigl\bigr
    \bigl\lvert#2\bigr\rvert
  \fi
}
\makeatother

\begin{document}

\(\abs{M} + \abs{M^2} + \abs{\abs{M^2}^2}\)

\(\abs{M} + \abs{M^2} + \bigl\lvert \abs{M^2}^2 \bigr\rvert\)

\( \abs{M^{2^{2^2}}_{f_{f_f}}} \)

\end{document}