Define a binary operator such that in displaymath subscripts are set below

At the moment I am using the following commands for tensor products respectively completed tensor products:

\documentclass{article}

\newcommand{\tensor}{\otimes}
\newcommand{\comptensor}{\mathbin{\hat{\otimes}}}

\begin{document}
$\testline$
$\textstyle \testline$
\end{document}


Both lines look like this:

I would like to change the definition of \tensor and \comptensor such that in textstyle nothing changes, but in displaystyle I would like the subscript to be printed below the operator instead of besides it.

With the natural syntax, so you can change your mind and avoid the underset symbol.

\documentclass{article}
\usepackage{amsmath}

\makeatletter
\newcommand{\tens}{\gen@tens{\otimes}}
\newcommand{\comptens}{\gen@tens{\hat{\otimes}}}
\newcommand{\gen@tens}[1]{%
\@ifnextchar_{\gen@@tens{#1}}{\mathbin{#1}}%
}
\def\gen@@tens#1_#2{%
\mathpalette\gen@@@tens{{#1}{#2}}%
}
\newcommand\gen@@@tens[2]{\mathbin{\gen@@@@tens#1#2}}
\newcommand\gen@@@@tens[3]{%
\ifx#1\displaystyle
\mathop{#2}\limits_{#3}%
\else
{#2}_{#3}%
\fi
}
\makeatother

\begin{document}

$A\tens B\comptens C \quad A\tens_{k}B\comptens_{k}C$
\begin{center}
$A\tens B\comptens C \quad A\tens_{k}B\comptens_{k}C$
\end{center}

\end{document}


• This is precisely what I had in mind :-) Although I don't quite understand why we need so many of the \gen@@...@tens. Now I only whish the spacing was handled correctly with LuaLaTeX (tex.stackexchange.com/questions/429355/…) but that's another story... – John Dorian May 1 '18 at 7:21

I did have to change the syntax just slightly, so that \comptensor takes an optional argument for its subscript, rather than using the underscore notation.

One cannot use the \limits style approach, because that only applies for the \mathop class, and not \mathbin. Thus, I used a stack in order to get the \displaystyle underset.

\documentclass{article}
\usepackage{stackengine}
\newcommand{\tensor}{\otimes}
\newcommand{\comptensor}[1][]{
\ifx\relax#1\relax\mathbin{\hat\otimes}\else\mathchoice
{\mathbin{\ensurestackMath{\stackunder[1pt]{\hat{\otimes}}{\scriptstyle#1}}}}
{\mathbin{\hat{\otimes}_{#1}}}
{\mathbin{\hat{\otimes}_{#1}}}
{\mathbin{\hat{\otimes}_{#1}}}
\fi
}
\newcommand{\testline}[1][]{$#1 V \tensor W \qquad V \tensor_k W$\par
$#1 V \comptensor W \qquad V \comptensor[k] W$}

\begin{document}
\testline[\displaystyle]

\testline
\end{document}