Bug? LaTeX misparses nested optional arguments [duplicate]

Question

LaTeX seems to be having problems handling nested optional arguments. I've included a minimal code example that generates the error:

\documentclass[12pt,oneside]{book}
\begin{document}

\newcommand{\termbad}[2][]{#2#1}
\newcommand{\termgood}[2][]{#2#1}

\newcommand{\up}[1]{#1}
\newcommand{\dd}[1][]{#1}

\termgood{\up{Tart}}{Pop}
\termbad[\up{Tart}]{Pop}    

\termgood{\dd[Tart]}{Pop}
\termbad[{\dd[Tart]}]{Pop}
%\termbad[\dd[Tart]]{Pop}


\end{document}

If you run this code, it works. If you remove the % in the second to last line, it fails with:

Runaway argument?

Tart{Pop} \par \par \end {document}

!File ended while scanning use of \dd.

After further tinkering, I have discovered the problem: LaTeX is reading the ] for the inner optional argument as the end of the outer optional argument and then passing a malformed argument in. So it is passing in \dd[Tart as the optional argument to \termbad instead of \dd[Tart].

This still feels like a bug to me. Nesting braces in arguments works, brackets should as well. The following are workarounds:

• Enclose the inner optional argument in braces (as in the line before %).

• Redefine your inner function to require its argument instead of using optional arguments.

• Redefine your outer function to require its argument instead of using optional arguments.

Any one of the three works around the issue.

(Reposted here; question was originally posted to stackexchange and then closed instead of being migrated.)

Answer 1

When TeX reads arguments, then TeX only checks for matching curly braces (characters with catcode 1 and 2). Square brackets are not special in this sense. The first ] that is not hidden inside curly braces is taken as the end of the optional argument. Therefore an additional set of braces is the usual solution:

\foo[{\bar[...]}]{...}

It is only a bug, if this does not work, e.g. if the definition passes optional arguments to other commands:

\def\foo#1#2{\bar[#1]{#2}}

This should be

\def\foo#1#2{\bar[{#1}]{#2}}

Then #1 may contain square brackets, especially ].