# Using \toks registers

I was reading this tutorial for TeX programming:

http://pgfplots.sourceforge.net/TeX-programming-notes.pdf

recommended in the 2-nd answer here: Where do I start LaTeX programming?

It says:

\toks<number> There are also 255 token registers which can be thought of as special string variables. Of course, every macro assignment \def\macro{ content } is also some kind of string variable, but token registers are special: their contents won’t be expanded when used with \the\toks number . This can be used for fine grained expansion control, see Section 2.3 below.

But no examples of using \toks registers. What does he mean: "special string variables"? For 2 previous kinds of registers I could assign the value:

\count0=12
\dimen0=1.102pt


And print it:

Value1: \the\count0
Value2: \the\dimen0


But I don't understand, how to use \toks.

-
Good question. My very limited (and probably wrong) understanding is that you store tokens into a token register with \toks<n>={nearly arbitrary stuff goes here} and then \the\toks<n> grabs the stored tokens and hands them off directly to TeX's stomach. –  kahen Apr 5 at 22:04
The assertion "2. The contents of token registers will never be expanded" is false. It's not expanded when \the\toks<number> is used in the replacement text of an \edef (or \xdef). –  egreg Apr 5 at 22:05
@kahen No; \the\toks<n> delivers the contents of the register to the input stream and expansion takes place normally (except in an \edef, where the expansion will take place when the \edefed macro will be used). –  egreg Apr 5 at 22:07
@kahen Ahh, yes. He doesn't say, that the string must be enclosed in figure brackets to be assigned to \toks register. Now I got it working. –  user4035 Apr 5 at 22:21

A token register is very similar to a macro with no arguments, but there are differences in syntax and expansion rules.

Using the plain and latex definition of \toks@ for \toks 0 you can store the tokens abc via

\toks@{abc}


or

\def\tmp{abc}


Note that a macro has a specific command to set the macro (\def or wrappers around that such as \newcommand) but token registers just use the register name followed by a _balanced text group.

Conversely unlike macros which expand just by referencing their name, token registers, like other registers are expandable and their contnts are accessed by expanding \the.

So

\the\toks@


and

\tmp


each expand to abc.

Apart from these syntactic differences the expansion of token registers differs from macros in two important ways.

1) expansion in \edef and other expansion only contexts is limited to one level/

\def\tmp{aaa}

\def\tmpb{xx \tmp}

\toks@{xx \tmp}

\edef\A{\tmpb}

\edef\B{\the\toks@}


Now \tmpb and \toks@ contain the same token list xx \tmp but \A is defined by fully expanding expandable tokens and so has defnition xx aaa but the tokens returned by the are not further expanded in the edef and so \B has definition xx \tmp

2) The second difference is that as token registers do not have arguments # does not need to be (and is not) special when defining or expanding the register.

\toks{#1}


is just a token register consisting of a list of two tokens # and 1.

This is put to use in LaTeX's \g@addto@macro macro.

\g@addto@macro@\foo{abc}


is supposed to add abc to the end of the current definition of \foo.

A simple (and if I recall correctly original) definition could be

\makeatletter

\gdef\foo{123}

\def\gaA#1#2{%
\expandafter\gdef\expandafter#1\expandafter{#1#2}}

\gaA\foo{abc}

\show\foo


That works fine and shows the definition as 123abc however try

\gaA\foo{#}


and you get

! Illegal parameter number in definition of \foo.


However we can use the fact that # isn't special in a toks register and that the register contents are only expanded once

\long\def\g@addto@macro#1#2{%
\begingroup
\toks@\expandafter{#1#2}%
\xdef#1{\the\toks@}%
\endgroup}


First the \toks@ register is defined to contain the expansion of the macro passed in the first argument, followed by the contents of the second argument. # is safe to use in a toks assignment. Then the macro is globally defined to be the expansion of \thetoks which is exactly the contents of \toks@ with no further expansion, even if that conatins # tokens:

\makeatletter

\gdef\foo{123}

\show\foo

\show\foo


produces

> \foo=macro:
->123abc.
l.7 \show\foo

?
> \foo=macro:
->123abc##.
l.11 \show\foo


where # has been added as intended (only one # has been added the doubled ## is an artefact of using \show.)

-
The fact that # is not special makes toks registers particularly valuable for saving "metacode", i.e. code that is involved in the generation of more code, rather than execution. Macro definitions, parts of macro definitions, and so on, as opposed to things that need to be expanded as-is. By contrast, the expansion-inhibiting property doesn't seem to be as big a deal after eTeX's \unexpanded. (This is addressed more to the audience, of course.) –  Ryan Reich Apr 6 at 0:03
What does the @ symbol after the register name mean? It is not mentioned in the book. I googled, but in vain. –  user4035 Apr 6 at 8:46
@user4035 -- the @ symbol in a command name means that this is an "internal" command, and to use it, it must either be within the scope of \makeatletter ... \makeatother, or in a .sty file where it is by default assumed to be of type "letter". –  barbara beeton Apr 6 at 15:08
@user4035 it's not after the name it is the command name toks@ (which is defined in latext o mean token register 0) –  David Carlisle Apr 6 at 16:04