# Is \toks necessary to define \g@addto@macro? Can token registers be avoided in generel?

In LaTeX2e kernel, there is a useful macro \g@addto@macro:

\long\def\g@addto@macro#1#2{%
\begingroup
\toks@\expandafter{#1#2}%
\xdef#1{\the\toks@}%
\endgroup}


However, I think the definition is rather complex. Why it is not defined as this?

\def\g@addto@macro#1#2{%
\expandafter\gdef\expandafter#1\expandafter{#1#2}} % or \xdef?


I cannot see any differences between the two implementation except that \par is allowed in the former one.

And we know that etoolbox package and LaTeX3 use another approach with help of eTeX:

% etoolbox
\newcommand{\expandonce}[1]{%
\unexpanded\expandafter{#1}}
\newrobustcmd{\appto}[2]{%
\ifundef{#1}
{\edef#1{\unexpanded{#2}}}
{\edef#1{\expandonce#1\unexpanded{#2}}}}

% LaTeX3
\cs_new:Npn \exp_not:o #1 { \etex_unexpanded:D \exp_after:wN {#1} }
\cs_new_protected:Npn \tl_put_right:Nn #1#2
{ \cs_set_nopar:Npx #1 { \exp_not:o #1 \exp_not:n {#2} } }


It seems that \toks is not that useful: I cannot find any use of \tex_toks:D, say \toks in source3 at all. So, do we really need \toks registers?

NOTE: I'm not asking a question about LaTeX (2e/3) kernel, \g@addto@macro is just an example. This is more a comparison between macro and token registers in TeX.

-
The 'rules' are very different depending on the existence of e-TeX. \g@addto@macro predates e-TeX, so the two parts to the question are rather separate here. –  Joseph Wright Dec 31 '11 at 18:44
@Joseph: Yes I know that. But I need only \expandafter to define \g@addto@macro. Of course \toks registers themselves may be useful without eTeX. –  Leo Liu Dec 31 '11 at 18:45
What I see is a better error recovery with the kernel definition. –  egreg Dec 31 '11 at 19:15
@egreg It's the # issue that's key here: I'd hate to need to double them in something like \AtBeginDocument! –  Joseph Wright Dec 31 '11 at 19:25
@LeoLiu toks can be passed around without passing them through TeX's reading apparatus. Why do you say that the definition is more complex? It is actually less code with any of the others. With toks you can build lists, have unions, append, prepend etc. and the code is normally very short. The other advantage of toks, they can be used for marks etc. –  Yiannis Lazarides Dec 31 '11 at 19:37

I'll take the answer in two parts, without e-TeX and with e-TeX, with a couple of subdivisions to the latter.

Without e-TeX, you do need token registers to be able to control expansion. Inside an \edef is the classic example, where

\toks@{\mymacro\myothermacro...}%
\edef\foo{\the\toks@}%


will not expand our macros. That's particularly required when you do not know what the input here might be: if it's known to be only one token then \noexpand would do. The same applies in other expansion contexts such as a \write.

(I'm assuming here that readers know that \the\toks@ will expand only to the content of the toks inside an \edef, and no further. Depending on the nature of \mymacro, etc., it may be critical to avoid exhaustive expansion.)

On the question of why use a toks rather than a simple \expandafter chain, the LaTeX2e sources give the game away:

% \begin{macro}{\g@addto@macro}
% Globally add to the end of a macro.
% \changes{v0.2w}{1994/01/31}
%     {Use toks register to avoid hash' problems}
% \changes{v1.0o}{1995/05/17}
%     {Make long for latex/1522}
% \changes{v1.0w}{1996/12/17}
%     {Use \cs{begingroup} to save making a mathord}
% \changes{v1.0x}{1997/02/05}
%     {missing percent /2402}
%    \begin{macrocode}
\begingroup
\toks@\expandafter{#1#2}%
\xdef#1{\the\toks@}%
\endgroup}
%    \end{macrocode}
% \end{macro}


Try \g@addto@macro\foo{\def\baz#1{}} or anything else containing a # without the use of a toks and observe the issue: you'd need to double all of the #s.

With e-TeX, the \unexpanded primitive is available. This acts like a toks but without the assignment, which allows a much simpler definition for \g@addto@macro along the lines of the two cited in the question. So in general there is little need to use a toks with e-TeX available: we dropped the entire l3toks module from LaTeX3 as a result.

TeX provides a few token parameters (for example \everypar) which act very much like toks and do have to be handled in a similar way. For LaTeX3, the plan is to provide a macro-like wrapper around all of them, so dealing with this unusual behaviour will be a job for the kernel team alone.

There is one place even with e-TeX where toks are useful, and that is when you want to refer to them by register number. The usual way of using a toks is

\newtoks\my@toks


and then use of the control sequence. However, you can also use toks by number

\toks0={tokens}


Normally, that's not a great approach. However, it does not consume a csname in the TeX hash table. So within a group (where we can be sure that we can freely use any register) it is potentially useful to use toks by number. Bruno Le Floch has taken advantage of this method in the l3regex` module for LaTeX3, as this avoids heavy use of the hash table for no benefit, and is a case where referencing by number is actually not such a bad idea.

-
Thank you. I've read some materials in The Advanced TeXbook. And your answer makes me clearer. –  Leo Liu Jan 1 '12 at 5:47