How does xparse handle nested delimited arguments?

Question

In this answer egreg mentions that the r argument type from the xparse package handles nested delimited arguments, as shown in this small example:

\documentclass{article}
\usepackage{xparse}

\NewDocumentCommand\foo{r()}{(#1)}

\begin{document}
\[ \foo(x\cdot\foo(y+\foo(z))\cdot w) \]
\end{document}

The package documentation doesn't explain how this is implemented, though. I tried looking at the code but it's really long and barely readable for someone who's not familar with LaTeX3.

As discussed in the question linked above, TeX's default delimited parameters can't be used for this. The only way I can think of is parsing through the whole following text via \futurelet, thereby somehow taking care that macros are expanded properly, until the final closing delimiter is found. Can anyone explain how xparse actually implements this?

Answer 1

In the basic approach, used by LaTeX2e, one looks for an optional [ using \futurelet (wrapped up in \@ifnextchar for LaTeX), the if found uses a delimited macro

\def\foo@aux[#1]{% Do stuff

For LaTex2e, that's it, but for xparse we then examine #1 to see if it contains a [. If it does, then the ] we grabbed isn't the right one to match the (now removed) [. Instead, we have to go though #1 and find the correct match. For example, with

\DeclareDocumentCommand\foo{o}{...}
\foo[[AAA]]]

the initial grab will match [[AA], which contains one [ so we have to find one more ]. A subtly to look out for is that something like

\foo[[a][b]]

requires multiple cycles to find everything.

The code in xparse does that using some macros which are named after the main command. That's done such that if the ] is missing, the resulting TeX error points back for the user to the document command, rather than some strange internal. The result is a slightly more complex set up in the code. (There are also some Bruno performance tricks and code to retain braces.)

---

For expandable grabbing, we can't use \futurelet, but that's true for all look-ahead: instead, we grab an argument and see if it is a single token matching the required look-ahead. Everything has to be set up in advance (no defining on the fly), so there is a bit more work ... but the basic idea remains the same.