Using macro with optional argument inside macro with optional argument

Question

I am trying to define a macro that has an optional argument that can take a macro with an optional argument as an argument. A minimal example to explain the problem more clearly:

\documentclass{article}

\newcommand*{\inn}[1][a]{#1}
\newcommand*{\out}[1][\inn]{\underline{#1}}
% \newcommand*{\out}[1][\inn]{\underline{#1}} % this also doesn't work (1)
% \newcommand*{\out}[1][\inn]{{#1}} % this also doesn't work (2)

\begin{document}
  $\out[\inn[b]]$
%  $\out[{\inn[b]}]$ % this works (3)
\end{document}

I get the error ! Argument of \\inn has an extra }. As indicated, I can fix the issue on a case-by-case basis, but not in general (using the right definition of the macros).

I first thought this was an issue caused by \underline being fragile. But using \ensuremath or \protected didn't lead me to a solution.

How can I fix the macros, so that they do what I want without errors?

Answer 1

The LaTeX2e approach to grabbing optional arguments uses a TeX delimited macro, which comes down to looking for an opening [ then using a macro which is defined in the form

\def\foo[#1]{...

That will always grab the first ] rather than do bracket matching. It is of course possible to match brackets. That is probably easiest to do using xparse as it is built-in

\documentclass{article}
\usepackage{xparse}
\NewDocumentCommand\inn{O{a}}{#1}
\NewDocumentCommand\out{O{\inn}}{\underline{#1}}

\begin{document}
  $\out[\inn[b]]$
\end{document}

Note that commands defined using xparse are declared using a different syntax to that uses by \newcommand. Here, an optional argument is type O with the default given as an argument to that O.

Answer 2

If you are open to xparse and its \NewDocumentCommand, the nesting of optional arguments without additional {} is possible.

The syntax of \NewDocumentCommand is different to \newcommand and for beginners a little bit confusing, but I switched lately to it and won't miss it any longer. I recommend xparse and its macros as long as no expansion is required (apart from the \DeclareExpandableDocumentCommand feature)

\documentclass{article}

\usepackage{xparse}
\newcommand*{\inn}[1][a]{#1}
\newcommand*{\out}[1][{\inn}]{\underline{#1}}
% \newcommand*{\out}[1][\inn]{\underline{#1}} % this also doesn't work
% \newcommand*{\out}[1][\inn]{{#1}} % this also doesn't work

\NewDocumentCommand{\Inn}{O{a}}{%
#1%
}

\NewDocumentCommand{\Out}{O{\Inn}}{%
\underline{#1}%
}


\begin{document}
\out[{\inn[b]}] % original version
  \Out[\Inn[b]] % Double []
  \Out[\Inn]
\end{document}