# Placement of limits

I'm trying to define a directed supremum operator in latex but I can't come up with something that would behave well in all situations. Here's what I've tried:

\documentclass{article}
\usepackage{amsmath}

\newcommand\dirup{\!{}^{\upset}}
\newcommand\upset{\ensuremath{\mathord{\uparrow}\mkern1mu}}

\begin{document}

$\mathop{\bigsqcup\dirup}_{i\in I}$
$$\mathop{\bigsqcup\dirup}_{i\in I}$$

$\mathop{\bigsqcup\dirup_{i\in I}}$
$$\mathop{\bigsqcup\dirup_{i\in I}}$$

$\mathop{\bigsqcup_{i\in I}}\dirup$
$$\mathop{\bigsqcup_{i\in I}}\dirup$$
\end{document}


In the first case, the subscripts are moved to right (in both math modes). In the second case, inline math mode works as expected but the display mode doesn't put the subscript below the supremum sign. Lastly the third case works as expected in display math mode but the inline math is broken because \dirup is moved to the right.

Any ideas how to fix it? I'm fine with redefining \dirup.

Edit: Note that the operator should also behave well when the subscript is omitted, e.g. the following shouldn't display the set \{ a \} too close to the supremum:

$\mathop{\bigsqcup\dirup} \{ a \}$


## 3 Answers

Shown, in textstyle, displaystyle, scriptstyle, and scriptscriptstyle. EDITED to consider case of no subscripts.

\documentclass{article}
\usepackage{amsmath}
\usepackage{stackengine,scalerel}
\def\foo{\,\ThisStyle{\ensurestackMath{%
\bigsqcup\stackengine{-0pt}{\!}{\SavedStyle\!^{\mathord{\uparrow}}}{O}{l}{F}{T}{S}}}\,}
\DeclareMathOperator*{\foobarX}{\foo}
\newcommand\foobar{\!\foobarX}
\begin{document}
\centering$a\foobar_{i\in I}a\qquad a\foobar a$
$a\foobar_{i\in I}a\qquad a\foobar a$
$\scriptstyle a\foobar_{i\in I}a\qquad a\foobar a$
$\scriptscriptstyle a\foobar_{i\in I}a\qquad a\foobar a$
\end{document}


Or maybe this slight variant, which was REEDITED to handle particular use cases provided by the OP. It screens for \{, [ and ( where spacing had previously been an issue.

\documentclass{article}
\usepackage{amsmath}
\usepackage{stackengine,scalerel}
\def\foo{\,\ThisStyle{\ensurestackMath{%
\bigsqcup\stackengine{-0pt}{\,}{\SavedStyle\!^{\mathord{\uparrow}}}{O}{l}{F}{T}{S}}}\!}
\DeclareMathOperator*{\foobarX}{\foo}
\makeatletter
\newcommand\foobar{\!\foobarX\@ifnextchar\{{\,}{\@ifnextchar[{\,}{\@ifnextchar({\,}{}}}}
\makeatother
\begin{document}
\centering$x\foobar_i A_i\qquad x\foobar_{i\in I}A_i\qquad x\foobar A_i$
$x\foobar_i A_i\qquad x\foobar_{i\in I}A_i\qquad x\foobar A_i$
$\scriptstyle x\foobar_i A_i\qquad x\foobar_{i\in I}A_i\qquad x\foobar A_i$
$\scriptscriptstyle x\foobar_i A_i\qquad x\foobar_{i\in I}A_i\qquad x\foobar A_i$

$\quad \foobar \{ A_i : i\in I \}\quad \foobar [ A_i : i\in I ]\quad \foobar ( A_i : i\in I )$
\end{document}


• Thank you! This is the closest to what I need so far! However, it still doesn't behave well when the subscript is omitted. (I've edited the question so that it's clearer what I mean by that) Is what I want even possible? Jul 21, 2016 at 10:22
• @TomasJakl Edited, but let me know what you think. Jul 21, 2016 at 10:30
• @StevenBSegletes It's almost perfect! All these are all my use cases and so they should always display well: $\foobar_i A_i \quad \foobar_{i\in I} A_i \quad \foobar \{ A_i : i\in I \}$ (The current version still keeps things a bit too close to each other in the third case). Jul 21, 2016 at 10:42
• @TomasJakl One way, without affecting the other horizontal settings (when followed by a subscript or a mathord) is to define \newcommand\foobar{\!\foobarX\@ifnextchar\{{\,}{\@ifnextchar[{\,}{\@ifnextchar({\,}{}}}} to specifically cover the cases when followed by \{, [, and (. Jul 21, 2016 at 10:49
• @TomasJakl Please see revision, showing the approach suggested above. Jul 21, 2016 at 10:59

Here's a complicated version, but, hey, it works!

The \negphantom macro can be found at Negative phantom inside equations

\documentclass{article}
\usepackage{amsmath,mathstyle}

\makeatletter
\newlength{\negph@wd}
\DeclareRobustCommand{\negphantom}[1]{%
\ifmmode
\mathpalette\negph@math{#1}%
\else
\negph@do{#1}%
\fi
}
\newcommand{\negph@math}[2]{\negph@do{$\m@th#1#2$}}
\newcommand{\negph@do}[1]{%
\settowidth{\negph@wd}{#1}%
\hspace*{-\negph@wd}%
}

\newcommand\bigsqcupwithup{%
\ifnum\mathstyle=\z@ % displaystyle
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
{%
\mathop{}\!\negphantom{{}^{\uparrow}}
\mathop{\hphantom{{}^\uparrow}{\bigsqcup}{}^{\uparrow}}%
}%
{\bigsqcup^{\scriptscriptstyle\uparrow}}%
}
\makeatother

\begin{document}

$\bigsqcupwithup_{i\in I}$
$A\bigsqcupwithup_{i\in I}B_i$
$A\bigsqcup_{i\in I}B_i$

\end{document}


A different version

\documentclass{article}
\usepackage{amsmath}

\makeatletter
\newlength{\negph@wd}
\DeclareRobustCommand{\negphantom}[1]{%
\ifmmode
\mathpalette\negph@math{#1}%
\else
\negph@do{#1}%
\fi
}
\newcommand{\negph@math}[2]{\negph@do{$\m@th#1#2$}}
\newcommand{\negph@do}[1]{%
\settowidth{\negph@wd}{#1}%
\hspace*{-\negph@wd}%
}

\DeclareRobustCommand{\bigsqcupwithup}{%
\@ifnextchar_{\sub@bigsqcupwithup}{\nosub@bigsqcupwithup}%
}

\newcommand{\nosub@bigsqcupwithup}{%
\mathop{
\mathchoice
{\disp@bigsqcupwithup{}}
{\nodisp@bigsqcupwithup{\hphantom{\scriptscriptstyle\uparrow}}}
{\nodisp@bigsqcupwithup{\hphantom{\scriptscriptstyle\uparrow}}}
{\nodisp@bigsqcupwithup{\hphantom{\scriptscriptstyle\uparrow}}}
}
}
\def\sub@bigsqcupwithup_#1{%
\mathop{
\mathchoice
{\disp@bigsqcupwithup{#1}}
{\nodisp@bigsqcupwithup{#1}}
{\nodisp@bigsqcupwithup{#1}}
{\nodisp@bigsqcupwithup{#1}}
}
}

\newcommand{\disp@bigsqcupwithup}[1]{%
\negphantom{{}^\uparrow}%
{\mathop{\hphantom{{}^\uparrow}{\bigsqcup}{}^\uparrow}\limits_{#1}}%
}
\newcommand{\nodisp@bigsqcupwithup}[1]{%
\bigsqcup^{\scriptscriptstyle\uparrow}_{#1}%
}
\makeatother

\begin{document}

$\bigsqcupwithup_{i\in I}X$
$\bigsqcupwithup X$
$A\bigsqcupwithup_{i\in I}B_i \qquad A\bigsqcupwithup B_i$

\end{document}


• Looks like magic to me but it works. Thanks! (However, it seems that mathstyle is incompatible with my LaTeX scripts) Jul 21, 2016 at 9:29
• @TomasJakl I feared that. :( This is a weak point of TeX, I'm afraid. With LuaTeX it would be easier. Jul 21, 2016 at 9:30
• Unfortunately, I can't use LuaTeX for submitting academic papers... Jul 21, 2016 at 9:44
• @TomasJakl Are you always using subscripts such as i\in I along with that symbol? With a syntax change it should be possible to cope with the problem in a way that doesn't use mathstyle. Jul 21, 2016 at 9:49
• No, I sometimes use it without. See my comment to Willie's solution. Jul 21, 2016 at 10:05

Since you are concerned with white space, maybe \mathrlap?

\documentclass{article}
\usepackage{amsmath}
\usepackage{mathtools}

\newcommand\dirup{\!{}^{\upset}}
\newcommand\upset{\mathrlap{\mathord{\uparrow}}}

\begin{document}

$\mathop{\bigsqcup\dirup}_{i\in I}$
$\mathop{\bigsqcup\dirup}_{i\in I}$

$\mathop{\bigsqcup\dirup_{i\in I}}$
$\mathop{\bigsqcup\dirup_{i\in I}}$

$\mathop{\bigsqcup_{i\in I}}\dirup$
$\mathop{\bigsqcup_{i\in I}}\dirup$
\end{document}


• Now, you're a 10k user -- use the moderator tools with care ;-)
– user31729
Jul 20, 2016 at 21:04
• Thanks! The first one seemed to solve the problem. It works quite well except that, for some reason, it gets too close to what follows when I omit the subscript, e.g. \mathop{\bigsqcup\dirup} I or \mathop{\bigsqcup\dirup} \{ a \}. Jul 21, 2016 at 9:15
• @TomasJakl: Ah! I hadn't realized you may use it without the subscript. In that case it is expected to fail since the code just makes the uparrow effectively invisible when space it being computed. So the subsequent text will be laid out as if no uparrow was there. The other answers are definitely better than mine! Jul 21, 2016 at 13:14
• @WillieWong That's fine, I should have specified it better. Thanks anyway. Jul 21, 2016 at 13:35