adaptive \ldots with no ("...") spacing

Question

Due to the typographic traditions I'm used to, I prefer my \ldots to have no spacing in between, ideally exactly the same amount as in text. In math mode I would like to write things like \(1, 2, \ldots, n\) and have the spacing be adaptive:

• if the next character is a comma, I'd like something very thin like \kern .08333em (i.e., half of a \thinspace) afterwards;
• if there is an opening parenthesis ("(", "[", "{", ...) before or a closing parenthesis (")", "]", "}", ...) after the dots, there should be no or only tiny additional spacing (i.e., less than \kern .08333em);
• and whatever other adaptations of space around \ldots or \dots are normally produced by (La)TeX;
• feel free to add other ideas.

What is the best way of redefining \ldots for these purposes?

Answer 1

\ldots is using \mathellipsis for the dots. It could be redefined to use the text version, but the mixup of text and math fonts might not always a good idea. \mathellipsis itself uses three dots as punctuation characters. That means, there is additional thin space between the dots. This can be changed by putting them into a subformula, then they are treated as \mathord atoms without additional space.

\documentclass[12pt]{article}
\usepackage{amsmath}
\renewcommand*{\mathellipsis}{%
  \mathinner{{\ldotp}{\ldotp}{\ldotp}}%
}
\begin{document}
$1,2,\ldots,3$
$(1,2,\ldots)$
\end{document}

Smaller spaces

Usually thin space is the smallest space in math mode, but to some degree \ldots can be adopted to smaller spaces:

\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage{letltxmacro}
\renewcommand*{\mathellipsis}{%
  \mathinner{{\ldotp}{\ldotp}{\ldotp}}%
}
\makeatletter
\@ifdefinable{\org@ldots}{%
  \LetLtxMacro\org@ldots\ldots
  \DeclareRobustCommand*{\ldots}{%
    \ifmmode
      \expandafter\my@ldots
    \else
      \expandafter\textellipsis
    \fi
  }%
}
\newcommand*{\neghalfmskip}{%
  \nonscript\mskip-.5\muexpr\thinmuskip\relax%
}
\newcommand*{\my@ldots}{%
  \mathellipsis
  \@ifnextchar,\neghalfmskip{%
  \@ifnextchar:\neghalfmskip{%
  \@ifnextchar;\neghalfmskip{%
  \@ifnextchar.\neghalfmskip{%
  \@ifnextchar!\neghalfmskip{%
  \@ifnextchar?\neghalfmskip{%
  \@ifnextchar){\mskip-.5\muexpr\thinmuskip\relax}{% negative kerning
  }}}}}}}%
}
\makeatother

\begin{document}
\noindent
$1,2,\ldots,3_{1,2,\ldots,3}$ (ldots)\\
$1,2,\mathellipsis,3_{1,2,\mathellipsis,3}$ (mathellipsis)\\
$(1,2,\ldots)_{(1,2,\ldots)}$ (ldots)\\
$(1,2,\mathellipsis)_{(1,2,\mathellipsis)}$ (mathellipsis)
\end{document}

TeX sets a thin space between inner atoms and punctuation chars in display and text style only, therefore the use of \nonscript. There is no space in case of the closing ), but because of the character shape, you probably want to have a negative kerning, this is applied in all math styles.

More generalized detection of a closing delimiter

The example uses Andrew's hint of \rightdelim@:

\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage{letltxmacro}
\renewcommand*{\mathellipsis}{%
  \mathinner{{\ldotp}{\ldotp}{\ldotp}}%
}
\makeatletter
\@ifdefinable{\org@ldots}{%
  \LetLtxMacro\org@ldots\ldots
  \DeclareRobustCommand*{\ldots}{%
    \ifmmode
      \expandafter\my@ldots
    \else
      \expandafter\textellipsis
    \fi
  }%
}
\newcommand*{\neghalfmskip}{%
  \nonscript\mskip-.5\muexpr\thinmuskip\relax%
}
\newcommand*{\my@ldots}{%
  \mathellipsis
  \@ifnextchar,\neghalfmskip{%
  \@ifnextchar:\neghalfmskip{%
  \@ifnextchar;\neghalfmskip{%
  \@ifnextchar.\neghalfmskip{%
  \@ifnextchar!\neghalfmskip{%
  \@ifnextchar?\neghalfmskip{%
    \rightdelim@
    \ifgtest@
      \mskip-.5\muexpr\thinmuskip\relax% negative kerning
    \fi
  }}}}}}%
}
\makeatother

\begin{document}
\noindent
$1,2,\ldots,3_{1,2,\ldots,3}$ (ldots)\\
$1,2,\mathellipsis,3_{1,2,\mathellipsis,3}$ (mathellipsis)\\
$(\neghalfmskip\ldots)_{(\neghalfmskip\ldots)}$ (neghalfmskip + ldots)\\
$(\ldots)_{(\ldots)}$ (ldots)\\
$(\mathellipsis)_{(\mathellipsis)}$ (mathellipsis)\\
$\{\ldots\}_{\{\ldots\}}$ (ldots)\\
$\{\mathellipsis\}_{\{\mathellipsis\}}$ (mathellipsis)
\end{document}

But for the detection of an opening delimiter before I do not see a way. There isn't a "\lastmathatom". Manually the space correction can be applied as the example file shows with \neghalfmskip.