\expandafter and \aftergroup: where do the 2n+1 and n^2-1 rules come from?

Question

Both TeX primitives \expandafter and \aftergroup have "rules of thumb" stating that

1. \expandafter should be used 2n+1 times to look n tokens ahead (is this correct?).
2. A similar "rule of thumb" holds for \aftergroup assignments: n^2-1.

Could anybody break down where these rules come from? I understand the basic idea of expanding a token to its definition, but I'm failing to generalize this to an arbitrary lookahead, that is, an expansion of the token n tokens ahead is still difficult for me.

PS: I've included an "aftergroup" tag, since I wasn't sure what tag would fit the best. Feel free to adjust if you think you know how to tag it correctly.

Answer 1

\expandafter is covered elsewhere so I'll restrict to \aftergroup.

Consider the plain tex file

{{{
xxx
\aftergroup a
xxx
\aftergroup\aftergroup
\aftergroup b
xxx
\aftergroup\aftergroup
\aftergroup\aftergroup
\aftergroup\aftergroup
\aftergroup c
xxx }
yyy }
zzz }

\bye

this produces

\aftergroup takes the following token out of the input stream and re-inserts it after the group ends.

So \aftergroup a inserts a after the first }

To understand the three \aftergroup before b consider them line by line as shown.

\aftergroup\aftergroup the first \aftergroup executes and places the (unexecuted) \aftergroup token at the end of the group (in this case it will be inserted after the a which had been previously placed in the \aftergroup queue.

\aftergroup b places b at the end of the current group, it will be inserted after the \aftergroup just discussed.

This means that at the first } the tokens a\aftergroup b are inserted, so a is typeset then \aftergroup removes b again and re-inserts it when the next } is encountered.

The seven \aftergroup before c are similar. Initially reading line you find that every other token is inserted after the current group, so after b the tokens \aftergroup\aftergroup\aftergroup c are inserted which as just discussed inserts \aftergroup c after that group, so finally at the last } c is inserted and gets typeset.

So as to lift a token one extra group level you need to aftergroup all the \aftergroup tokens used at the previous level, plus an \aftergroup on the token of interest, you find that you usually have 1,3,7,...2^n-1 \aftergroup tokens to lift a token through n group levels.

---

Note that it isn't a rule that there has to be 2^n-1 \aftergroup interesting effects can be obtained by other groupings:

{{{
xxx
\aftergroup\aftergroup a
yyy}zzz
\aftergroup\aftergroup\aftergroup\aftergroup b
www}qqq}mmm

\bye

produces

Here \aftergroup\aftergroup places \aftergroup after the first }; the a is not affected at all, so typesets as normal after xxx. Then at the } the \aftergroup token is inserted and applies to the first z so only two zz get typeset at that point, the first z is typeset at the next } after www.

The four \aftergroup before b are similar. Two \aftergroup tokens are removed and inserted at the next } and b is not affected at all, so typesets after the zz. the two \aftergroup are inserted at the next } so the input is

\aftergroup\aftergroup qqq}mmm

this inserts an \aftergroup after the } so is equivalent to

qqq}\aftergroup mmm

so one m is taken out of the input and two mm are typeset. The first m just silently vanishes as it would appear at the end of the current group but this is the top level so it is just lost (not an error).

Answer 2

The (unexpandable) primitive \aftergroup applies to the next token only; it makes TeX save the token in a FIFO list that will be delivered as soon as the current group ends. So

\begingroup\aftergroup x\aftergroup y\endgroup

is just the same as typing xy: the list is “first in, first out”. Note that in case of a macro, TeX only saves the macro, not its meaning. Thus

\def\foo{X}
\begingroup\def\foo{Y}\aftergroup\foo\endgroup

will print X, because the redefinition of \foo disappears as soon as \endgroup is executed.

It doesn't matter what group we're in; it can be any of the sixteen different group types. For instance, \hbox{A\aftergroup B} will box an A and print B outside the box; the first example could have been {\aftergroup x\aftergroup y}.

Leaving aside problems due to the limited size reserved for the FIFO list, what tokens can follow \aftergroup? Any token, even { or }; the definition of the LaTeX environment lrbox is very instructive in this respect.

If you want to chain tokens in order to reappear after the current group has ended, just precede each one by \aftergroup:

\begingroup\aftergroup\mbox\aftergroup{\aftergroup X\aftergroup}\endgroup

will make TeX see the tokens \mbox{X} just after the group has ended (and all settings are restored).

Note that \aftergroup{\mbox{X}} is not equivalent to the code above: \aftergroup does not accept arguments in braces, it only applies to the token that immediately follows it.

If you want to defer code when two groups end, it's easy:

\begingroup\begingroup
\aftergroup\aftergroup\aftergroup\foo
\endgroup\endgroup

The meaning should be clear: when the current group (at nesting level 2) ends, TeX will deliver \aftergroup\foo, which will save back \foo in the aftergroup list for deliver after the outer group ends. Any group level has its own incarnation of this list.

If you want to climb three levels, just remember that each group consumes one \aftergroup: so

\begingroup\begingroup\begingroup
\aftergroup\aftergroup
\aftergroup\aftergroup
\aftergroup\aftergroup
\aftergroup\foo
\endgroup\endgroup\endgroup

will do, because when the level 3 group ends, the list will contain

\aftergroup\aftergroup\aftergroup\foo

Do yourself the proof by induction that, in order to climb up n levels, you need 2ⁿ - 1 \aftergroup tokens before \foo.