# How does one append material to a token list?

Several times, the floatrow package uses \newtoks which (according to this website) means that a token register is allocated. An example code snippet is

\@ifdefinable\FR@everyfloat{\newtoks\FR@everyfloat}
\FR@everyfloat={\let\@footnotetext\@mpfootnotetext
\def\@mpfn{mpfootnote}\def\thempfn{\thempfootnote}\c@mpfootnote\z@
\floatobjectset\floatfont}


Suppose I want to add something (e.g., \setlength{\footnotesep}{0.2\baselineskip}) to the definition of \FR@everyfloat. One way I successfully applied in this answer is to overwrite the original definition, i.e., write

\FR@everyfloat={\let\@footnotetext\@mpfootnotetext
\def\@mpfn{mpfootnote}\def\thempfn{\thempfootnote}\c@mpfootnote\z@
\floatobjectset\floatfont
\setlength{\footnotesep}{0.2\baselineskip}}


However, I'd like to avoid duplicating the whole token list content; instead, I want to append material to the list the same way one would use \g@addto@macro (or \appto from the etoolbox package). However, my naïve approach to write

\g@addto@macro{\FR@everyfloat}{\setlength{\footnotesep}{0.2\baselineskip}}


doesn't work -- obviously, token lists must be treated different from macros.

So: How does one append material to a token list?

-

\documentclass{article}

\newtoks\abc
\newcommand{\appendtotoks}[2]{% #1=toks register, #2-toks to append
#1=\expandafter{\the #1#2}%
}

\begin{document}

\abc={A. B. C.}
\the\abc

\appendtotoks{\abc}{X. Y. Z.}
\the\abc

\end{document}

-

Token register expand to their content only if prefixed with \the, otherwise on their own they are not expandable, IIRC. You need to use \expandafter to expand the token register before assigning it to the register again:

Appending works like this:

\mytoks\expandafter{\the\mytoks<new code>}


Here you don't need another \expandafter at the beginning because token register expand the following tokens in their search for a balanced { } list.

Prefixing code is more complicated because you have to jump over it. This is normally done using a second token register:

\temptoks{<new code>}
\mytoks\expandafter{\the\expandafter\temptoks\the\mytoks}


This uses the fact that \the is expanding the next tokens as well to find a register.

-
This should just be \mytoks\expandafter{\the\expandafter\temptoks\the\mytoks} –  user9588 Jan 7 '12 at 11:07
@DavidKastrup: Right, I overlooked that. I will edit the answer. –  Martin Scharrer Jan 7 '12 at 11:34

WARNING: as Joseph Wright pointed out in the comments section, this won't work in newer versions of expl3.

And using LaTeX3 (man, this is my first try of it, so forgive (and correct) me if I get something wrong!):

\documentclass{article}

\usepackage{expl3}

\begin{document}

\ExplSyntaxOn
\toks_new:N\abc_toks
\toks_set:Nn\abc_toks{A. B. C.}
\toks_use:N\abc_toks
\par

\toks_put_right:Nn\abc_toks{X. Y. Z.}
\toks_use:N\abc_toks
\ExplSyntaxOff

\end{document}

-
This used to work, but won't with the current release of expl3. We removed the toks module a while ago, as it turns out not to be needed for writing LaTeX3 code. There, we define a 'token list variable' of the form \edef\myvar{\unexpanded{<stuff>}}. This can be used to hold any tokens, including #, and the various e-TeX based expansion controls we have mean that we also don't need the odd behaviour of toks for preventing expansion. So it was clearer over all to drop toks entirely. That does leave some awkwardness when working with LaTeX2e code: sorry about that! –  Joseph Wright Dec 13 '11 at 8:37
Thanks for pointing that. (I have a not-so-new version of expl3 and source3.pdf on my computer.) Sorry for confusion. (I did not delete this answer, since your comment does make all this useful for people who used expl3 and wonder why their code stopped working;). If you think it should be deleted, let me know (or delete it).) –  mbork Dec 13 '11 at 10:47
@JosephWright What if somebody wants token registers to play with them? :) Will there be the Thought Police popping out in case of crimes against LaTeX3? ;-) –  egreg Dec 13 '11 at 11:52
@egreg Bruno is using toks 'directly' (by number) for the l3regex module as it keeps down csname usage, etc., for this very heavy use case. What there isn't is a native LaTeX3 allocation function for toks. As I've indicated, there is no general reason to use them, and dropping support avoids the need for complex 'Do I use a tl or a toks?' discussion. –  Joseph Wright Dec 13 '11 at 11:55

Looking at mbork's solution and asking myself "Why isn't this included in the LaTeX kernel?", I discovered that it actually is. A search for #1=\expandafter{\the #1#2} didn't return anything, but a search for #1\expandafter{\the#1#2} (note the missing equal sign and space) returns (citing source2e, chapter 27)

We need a macro to add tokens to a hook.

485 \long\def\addto@hook#1#2{#1\expandafter{\the#1#2}}

This is TeX syntax, but otherwise identical to mbork's code. So the answer to my question is: Use \addto@hook.

-
Ha, I've also looked into the source, but didn't find it (I just scanned through the ToC). This is a very good point! Notice that the equal sign in assignments in TeX is usually optional (maybe except for things like \let\equals== ;) ). And please don't call this "mbork's" solution; first, Martin's answer is better, second, I remeber reading it somewhere (probably in the TeXbook). –  mbork Dec 12 '11 at 22:20
@mbork: I need to seriously delve into the TeXbook. –  lockstep Dec 12 '11 at 22:21
Yes, it is a very nice read. –  mbork Dec 13 '11 at 6:55
I forgot to add it to my answer. :( –  egreg Dec 13 '11 at 7:44

As others have pointed out, \the\mytoks delivers the contents of the token register \mytoks. When one wants to append something to a token register, the usual way is

\mytoks=\expandafter{\the\mytoks <tokens>}


Such code needs an explanation. A token register's name (not following \the) starts an assignment to that token register. An optional = can follow, and then there must be {<balanced text><right brace>. Every time a required token list has to begin with { (in the simbology of the TeXbook this denotes an explicit token of category 1 or a token \let to it) TeX expands the tokens it finds in order to come to {. Thus, in

\mytoks=\expandafter{\the\mytoks <tokens>}


\expandafter is expanded, delivering the contents of \mytoks and so the assignment can be performed. The assignment is terminated by an explicit right brace. A general macro for this can be

\long\def\appendto#1#2{\the#1#2}


where #1 is the name of the token register and #2 the token list to append.

A bit more complicated is to prepend something to the contents of a token register. Let's try; parameter #1 will denote a token register, #2 will be the tokens to be prepended:

\long\def\prependto#1#2{\toks0={#2}%
#1=\expandafter{\the\toks0\the#1}}


Here we're using a dirty trick: \the wants to know what it must act on; it sees \toks which has to be followed by a ; 0 starts a number, but TeX continues to expand to see if other digits follow. So the second \the is expanded.

Limitation: one can't \prependto the token register \toks0.

Is there a way to overcome this limitation? Yes, there is:

\long\def\prependto#1#2{\def\tempa{#2}%
\edef\tempb{\noexpand\tempa\the#1}%
#1=\expandafter\expandafter\expandafter{\tempb}}


Suppose we have \mytoks={\A\B\C} and that we call

\prependto\mytoks{\a\b\c}


Then TeX executes \def\tempa{\a\b\c}; we now exploit the fact that the token list resulting from \the\mytoks is not further expanded in an \edef, so this is equivalent to having said

\def\tempb{\tempa\A\B\C}


The next instruction is

\mytoks=\expandafter\expandafter\expandafter{\tempb}


which becomes

\mytoks=\expandafter{\tempa\A\B\C}


and, finally,

\mytoks={\a\b\c\A\B\C}


Of course, the first method is much more efficient. Instead of \toks0 one can use a permanently reserved token register, say \reservedtoks:

\def\prependto#1#2{\reservedtoks={#2}%
#1=\expandafter{\the\expandafter\reservedtoks\the#1}}


(exercise: find out why it works). Users wanting to say \prependto\reservedtoks{abc} are on their own. :)

A trickier strategy, that avoids temporary assignments, has been suggested by Bruno Le Floch:

\def\prependto#1{\expandafter\prependtoaux\expandafter{\the#1}#1}
\long\def\prependtoaux#1#2#3{#2{#3#1}}


Let's see with the same setting as before what happens with \prependto\mytoks{\a\b\c} (each line is what results from the expansion of the preceding one):

\prependto\mytoks{\a\b\c}
\expandafter\prependtoaux\expandafter{\the\mytoks}\mytoks{\a\b\c}
\prependtoaux{\A\B\C}\mytoks{\a\b\c}
\mytoks{\a\b\c\A\B\C}


(the = after \mytoks is optional).

Notes

1. The expansion to find an open brace don't happen for \def because the left brace delimiting the replacement text must be explicit.

2. \mytoks=\bgroup abc} is a valid assignment; on the contrary, \mytoks=\bgroup abc\egroup isn't: TeX will continue to scan tokens until finding an explicit (unbalanced) right brace.

3. \long is needed because the token list to append or prepend could contain \par (thanks to Bruno Le Floch for pointing it out).

-
my feeling is that all your \def should be \long\def. Also, you can do \long\def\prependto#1{\expandafter\prependtoaux\expandafter{\the#1}#1} and \long\def\prependtoaux#1#2#3{#2{#3#1}}` to avoid assignments. –  Bruno Le Floch Dec 13 '11 at 0:14
Seriously, TeX.SE should allow for double upvotes. –  mbork Dec 13 '11 at 6:58