# \sideset with parentheses

I am now trying to define some "decorated operators" similar to in several ways and I want to find my preference on decorating operators.

A simple "prepared" way is to use the command \sideset of amsmath, though the position of the limit-style subscript may depend on one's preference. Some month before, my friend told me a confliction of \sideset with parenthesis, e.g. the code

$\left|\sideset{}{^{\smash{\sharp}}}\sum_{x<n\le x+y}\chi(n)a(n)\right|$


gives so the limit-style subscript x<n\le x+y and the modulus sign \left| clash each other.

I thought that this is because \sideset turns math operators to ordinary math atoms. Therefore, I tried to use \mathop as follows:

$\left|\mathop{\sideset{}{^{\smash{\sharp}}}\sum} _{x<n\le x+y}\chi(n)a(n)\right| \quad \left|\mathop{\sideset{}{^{\smash{\sharp}}}\sum _{x<n\le x+y}}\chi(n)a(n)\right|$


this gives

I am still wondering why \sideset behaves not well with some parentheses and why the above two applications of \mathop give so different results. For me (as a beginner of LaTeX), the latter one of my \mathop applications seems more likely to avoid the clash, but the truth was opposite...

Thus, I would like to ask:

1. What is the reason of the above clash of \sideset and parentheses? Is my usage of \sideset wrong? Or is this a "bug" of \sideset?
2. If this is a bug of \sideset, what is a good way to avoid it?
3. Why my two applications of \mathop above give different results?

Note: According to egreg's answer, I found the above clash happens since \sideset shift math operator "too much" if the limit-style subscript is too long. Now we let \egregsideset be the corrected version of \sideset due to egreg. Then

$\fbox{\displaystyle% \sideset{}{^\text{quite very very very long}}\sum% } \fbox{\displaystyle% \egregsideset{}{^\text{quite very very very long}}\sum% }$


gives

which although I am not sure which one is better, but I guess the original \sideset is more natural in this aspect. Thus, I add a new question

1. How can we modify \egregsideset with avoiding the above difference with \sideset?

It is a bug of \sideset. If I try

$\left|\sideset{}{}\sum\right|\showlists$


I get

\mathinner
.\left"26A30C
.\mathord
..\hbox(0.0+0.0)x0.0
.\mathop
..\kern 0.0
..\mathord
...\hbox(10.50006+5.50006)x0.0
....\mathon
....\vbox(10.50006+5.50006)x0.0
....\mathoff
..\mathord
...\hbox(10.50006+5.50006)x14.44447
....\mathon
....\hbox(1.0+15.00012)x14.44447, shifted -9.50006
.....\OMX/cmex/m/n/10 X
....\mathoff
.\right"26A30C


so essentially Open Ord Op Close. This in contrast to \left|\sum\right| that shows

\mathinner
.\left"26A30C
.\mathop
..\fam3 P
.\right"26A30C


so Open Op Close. A visual comparison

shows quite a different problem than yours: the summation is too much on the right.

However, if I change the definition of \sideset to

\makeatletter
\renewcommand{\sideset}[3]{%
\@mathmeasure\z@\displaystyle{#3}%
\global\setbox\@ne\vbox to\ht\z@{}\dp\@ne\dp\z@
\setbox\tw@\box\@ne
\@mathmeasure4\displaystyle{\copy\tw@#1}%
\@mathmeasure6\displaystyle{#3\nolimits#2}%
\mathop{\hbox to\dimen@{}}\!% <--- was only \hbox to\dimen@{}
\mathop{\kern-\dimen@\box4\box6}%
}
\makeatother


so TeX will see a \mathop before doing any operation and the thin space between the two consecutive Op atoms is nullified by \!, I get the expected result:

\documentclass{article}
\usepackage{amsmath}

\makeatletter
\renewcommand{\sideset}[3]{%
\@mathmeasure\z@\displaystyle{#3}%
\global\setbox\@ne\vbox to\ht\z@{}\dp\@ne\dp\z@
\setbox\tw@\box\@ne
\@mathmeasure4\displaystyle{\copy\tw@#1}%
\@mathmeasure6\displaystyle{#3\nolimits#2}%
\mathop{\hbox to\dimen@{}}\!%
\mathop{\kern-\dimen@\box4\box6}%
}
\makeatother

\begin{document}
\begin{gather*}
\left|\sideset{}{}\sum\right|%\showlists
\\
\left|\sum\right|
\\
\left|\sideset{}{^{\smash{\sharp}}}\sum_{x<n\le x+y}\chi(n)a(n)\right|
\\
\left|\sum_{x<n\le x+y}\chi(n)a(n)\right|
\end{gather*}

\end{document}


I'm not really sure this can be fixed, because it might change existing documents.

However, the implementation of \sideset is buggy to begin with and you can test it with

\documentclass{article}
\usepackage{amsmath}
\begin{document}
$\left|\sideset{}{^\text{quite very very very long}}\sum_{aaaaaaaaaaaaaaaaaa}\right|$
\end{document}


(original \sideset).

I can offer a different implementation:

\documentclass{article}
\usepackage{amsmath}
\usepackage{xparse}

\makeatletter
\NewDocumentCommand{\xsideset}{mmme{_^}}{%
\mathop{%
% half width of #3
\settowidth{\dimen0}{$\m@th\displaystyle#3$}%
\dimen0=.5\dimen0
% half width of #3 with subscripts or superscripts,
% half width of #3 removed
\settowidth{\dimen2}{$% \m@th\displaystyle#3% \IfValueT{#4}{_{#4}}% \IfValueT{#5}{^{#5}}%$}%
\dimen2=.5\dimen2
% prescripts
\sbox6{\scriptspace\z@$\displaystyle{\vphantom{#3}}#1$}
% postscripts
\sbox8{\scriptspace\z@$\displaystyle{\vphantom{#3}}#2$}
% typeset the thing
\ifdim\wd6>\dimen2 \kern\dimexpr\wd6-\dimen2\relax\fi
{%
\mathop{\llap{\copy6}{\displaystyle#3}\rlap{\copy8}}\limits
\IfValueT{#4}{_{#4}}%
\IfValueT{#5}{^{#5}}%
}%
\ifdim\wd8>\dimen2 \kern\dimexpr\wd8-\dimen2\relax\fi
}%
}
\makeatother

\begin{document}
\begin{gather*}
\left|\sum\right|\\
\left|\xsideset{}{}\sum\right|\\
\left|\xsideset{}{^{\smash{\sharp}}}\sum_{x<n\le x+y}\chi(n)a(n)\right|\\
\left|\xsideset{}{^\text{quite very very very long}}\sum\right|\\
\left|\xsideset{^{\text{quite very very very long}}}{}\sum_{\text{not quite so long}}\right|\\
\end{gather*}
\end{document}


• +1 Crazy as usual :). I was brave enough to try to improve the question. If that's not ok then just let me know. I left the original image for now so that you see the change/contribution I made. – Dr. Manuel Kuehner Jan 28 '18 at 22:36
• Thanks very much again, egreg! The answer went far beyond of my knowledge, so it took a time to reply. Your solution is very nice and educative, but if \dimen@ take negative value at \mathop{\hbox to\dimen@{}}\! (which was the reason of the clash in my question), then some excess space will appear on the left side of \sideset since negative adjustment is cancelled by \mathop. Are there some trick to avoid this? (Conditional branches?) – Yuta Suzuki Jan 30 '18 at 7:20
• > Dr. Mauel Kuehner, thanks for improvement, there is no problem to improve the question too! – Yuta Suzuki Jan 30 '18 at 7:22
• @YutaSuzuki I'm not sure I understand the objection. Using \hbox to\dimen@ inside \mathop doesn't change the typesetting except possibly for adding a thin space in front, which is the missing thing in the example and in general. – egreg Jan 30 '18 at 7:31
• @YutaSuzuki Yes, I see. I guess there's something wrong in the logic of the computation. I'll try and work on it. – egreg Jan 30 '18 at 7:54