The following example measures the formula inside \abs
with \left
/\right
and with \bigl
/\bigr
. If the version \left
/\right
is larger, then the version with \bigl
/\bigr
is used.
\documentclass{article}
\usepackage{amsmath}
\usepackage{mleftright}
\makeatletter
\newcommand*{\abs}[1]{%
\mathpalette\@abs{#1}%
}
% Version with the horizontal spacing of `\left` and `\right` in all cases:
% \newcommand*{\abs}[1]{%
% \mathinner{\mathpalette\@abs{#1}}%
% }
\newcommand*{\@abs}[2]{%
\sbox0{$\m@th#1\mleft\lvert#2\mright\rvert$}%
\sbox2{$\m@th#1\bigl\lvert#2\bigr\rvert$}%
\ifdim\wd0>\wd2 %
\bigl\lvert#2\bigr\rvert
\else
\ifdim\ht0>\ht2 %
\bigl\lvert#2\bigr\rvert
\else
\ifdim\dp0>\dp2 %
\bigl\lvert#2\bigr\rvert
\else
\mleft\lvert#2\mright\rvert
\fi
\fi
\fi
}
\makeatother
\begin{document}
\(\abs{M} + \abs{M^2} + \abs{\abs{M^2}^2}\)
\(\abs{M} + \abs{M^2} + \bigl\lvert \abs{M^2}^2 \bigr\rvert\)
\end{document}

Remarks:
Instead of \left
/\right
macros \mleft
/\mright
of package mleftright
are used to avoid the additional spacing of \left
/\right
.
The method is not "bulletproof". It makes the following assumption. If the delimiters, picked by \mleft
and \mright
, are larger than the version with \bigl
and \bigr
, then this case can only be detected, if at least either the width, height or depth increases. For the case that the width remains the same, then the larger delimiters need to be slightly larger than the formula, otherwise the formula would hide the height/depth of the delimiters. I haven't analyzed all resizeable delimiters, but I think, usually the width of the delimiter will become larger that is easily detected.
Second approach
This approach implements the algorithm of minimal delimiter size according to "The TeXbook" by D. E. Knuth, "Appendix G: Generating Boxes from Formulas", item "19.":
If the math list begins and ends with boundary items, compute the
maximum height h and depth d of the boxes in the translation of
the math list [...]. Let a = σ22 be the axis
height, and let δ = max(h-a, d+a) be the
amount by which the formula extends away from the axis. Replace the
boundary items by delimiters whose height plus depth is at least
max(⎣δ/500⎦f, 2δ-l), where f is the
\delimiterfactor
and l is the \delimitershortfall
. [...]
It can deal correctly with an expression like M^{2^{2^2}}_{f_{f_f}}
, which trips up the first approach by violating its assumption about delimiter dimensions.
Example file:
\documentclass{article}
\usepackage{amsmath}
\usepackage{mleftright}
\makeatletter
\newcommand*{\abs}[1]{%
\mathpalette\@abs{#1}%
}
% Version with the horizontal spacing of `\left` and `\right` in all cases:
% \newcommand*{\abs}[1]{%
% \mathinner{\mathpalette\@abs{#1}}%
% }
\newcommand*{\@abs}[2]{%
% a := math axis height -> \dimen0
\sbox0{$#1\vcenter{}$}%
\dimen0=\ht0 %
% formula without delimiters -> \box0
% h := height of formula -> \ht0
% d := depth of formula -> \dp0
\sbox0{$#1#2$}%
% delta := max(h-a, d+a) -> \dimen2
\dimen2=\dimexpr\ht0-\dimen0\relax
\dimen4=\dimexpr\dp0+\dimen0\relax
\ifdim\dimen4>\dimen2 %
\dimen2=\dimen4 %
\fi
% delimiter's total height >= max(floor(delta/500)*f, 2*delta - l)
% -> \dimen4
% f := \delimiterfactor
% l := \delimitershortfall
\dimen4 = \dimen2 %
\divide\dimen4 by 500 %
\dimen4=\delimiterfactor\dimen4 %
\dimen6=\dimexpr 2\dimen2 - \delimitershortfall\relax
\ifdim\dimen6>\dimen4 %
\dimen4=\dimen6 %
\fi
% formula with \bigl/\bigr
\sbox0{$#1\bigl\lvert\bigr\rvert$}%
% comparison: Use \mleft/\mright, if \bigl/\bigr is larger than needed
\ifdim\dimexpr\ht0+\dp0\relax>\dimen4 %
% Case 1: \bigl/\bigr is larger than the minimum needed height.
% This means that either (\lvert,\rvert) or (\bigl\lvert,\bigr\rvert)
% provide the smallest delimiter size that meet the minimum height requirement.
% Consequently we can have \mleft/\mright choose (they won't pick anything
% larger than \bigl/\bigr).
\mleft\lvert#2\mright\rvert
\else
% Case 2: \bigl/\bigr is exactly right or smaller than the required minimum height
% => use \bigl\bigr
\bigl\lvert#2\bigr\rvert
\fi
}
\makeatother
\begin{document}
\(\abs{M} + \abs{M^2} + \abs{\abs{M^2}^2}\)
\(\abs{M} + \abs{M^2} + \bigl\lvert \abs{M^2}^2 \bigr\rvert\)
\( \abs{M^{2^{2^2}}_{f_{f_f}}} \)
\end{document}
