# smashoperator overlaps big parentheses with limits

\smashoperator pulls big parentheses too close causing them to overlap with the limits. Is there any workaround?

Edit: An ad-hoc solution is to force an appropriate amount of empty space to the right/left of the left/right parentheses using \, \quad etc. But these "tricks" obviously defeat the whole point of using LaTeX.

Example:

\documentclass[american]{article}
\usepackage{amsmath,amssymb,amsthm,mathtools}
\usepackage{babel}

\begin{document}

$\lambda\left(\smashoperator{\bigcup_{x=lowerlimit}^{upperlimit}} f(x) \right)$

\end{document}

-
\smashoperator does precisely what it should do here. What result did you expect? (Maybe try \smashoperator[r]?) –  Andrey Vihrov Apr 8 '11 at 12:09
I expected that only f(x) would get pulled close and that the parentheses would stay outside the limits. I believe that is a reasonable expectation. And although with the current example, \smashoperator[r] luckily seems to avoid overlap, if you change the limits to "x=longlowerlimit" and "longupperlimit" then the overlap again occurs. –  user4522 Apr 8 '11 at 12:54
It does what it does, how should \smashoperator know there is a fence behind it? –  daleif Apr 8 '11 at 15:30
I'm no (La)TeX expert, but I think it should be possible to modify \smashoperator to consider this case. I think the creators never considered this possibility. –  user4522 Apr 8 '11 at 15:44
Not unless someone provide it. There will be a lot of special cases to consider, so it would be a lot easier to use a different solution in the cases where \smashoperator does not fit. –  daleif Apr 8 '11 at 16:09

As noted in the comments, it's very difficult to define a \smashoperator command that deals with all possible cases involving long limits. To understand why it is difficult, you need to know a little about boxes in TeX. Roughly speaking, horizontally smashed boxes (the limits in this case) are considered to have zero width, and there is no easy way to say

Small objects should consider the limits to be smashed, but large objects should not.

which is what you need.

The following code seems to work in your case, but will probably fail in others that I have not considered. In my opinion it would be better to avoid long limits, because they are grotesque (perhaps this is why the author(s) of mathtools did not provide an easy way to construct them).

\documentclass{minimal}
\usepackage{mathtools}
\newcommand{\test}[4]{ %
%
\newbox\bigbox
\newbox\smallbox
\newbox\argbox
\newdimen\smallwidth
\newdimen\offsetwidth
\newdimen\argwidth
%
%Store the unsmashed operator in \bigbox.
\sbox\bigbox{\ensuremath{\displaystyle #1_{#2}^{#3}}}
%Now store the smashed operator in \smallbox.
\sbox\smallbox{\ensuremath{\smashoperator{#1_{#2}^{#3}}}}
%Finally, store the argument in \argbox
\sbox\argbox{\ensuremath{\displaystyle \,#4}}
%Calculate the width of the argument
\argwidth=\wd\argbox
%
%Calculate the width by which the limits protrude. Note:
%this will evaluate to zero if the limits do not protrude.
\offsetwidth=\wd\bigbox
\smallwidth=\wd\smallbox
\multiply\smallwidth by -1
\divide\offsetwidth by 2
%
%Place the unsmashed operator and the argument on the page,
%pulling back the argument by the appropriate distance.
\usebox\bigbox\hspace{-\offsetwidth}\usebox\argbox
%
%If the width of the argument is less than the width of
%the protrusion, we now need to add some more space.
\ifnum\offsetwidth>\argwidth \hspace{\offsetwidth}\hspace{-\argwidth} \fi
%
}
\begin{document}
\begin{equation*}
\left[\test{\bigcup}{longlimit}{longlimit}{f(x)} \right]