Take the 2-minute tour ×
TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. It's 100% free, no registration required.

I was reading this tutorial for TeX programming:

http://pgfplots.sourceforge.net/TeX-programming-notes.pdf

recommended in the 2-nd answer here: Where do I start LaTeX programming?

It says:

\toks<number> There are also 255 token registers which can be thought of as special string variables. Of course, every macro assignment \def\macro{ content } is also some kind of string variable, but token registers are special: their contents won’t be expanded when used with \the\toks number . This can be used for fine grained expansion control, see Section 2.3 below.

But no examples of using \toks registers. What does he mean: "special string variables"? For 2 previous kinds of registers I could assign the value:

\count0=12
\dimen0=1.102pt

And print it:

Value1: \the\count0
Value2: \the\dimen0

But I don't understand, how to use \toks.

share|improve this question
1  
Good question. My very limited (and probably wrong) understanding is that you store tokens into a token register with \toks<n>={nearly arbitrary stuff goes here} and then \the\toks<n> grabs the stored tokens and hands them off directly to TeX's stomach. –  kahen Apr 5 '13 at 22:04
    
The assertion "2. The contents of token registers will never be expanded" is false. It's not expanded when \the\toks<number> is used in the replacement text of an \edef (or \xdef). –  egreg Apr 5 '13 at 22:05
    
@kahen No; \the\toks<n> delivers the contents of the register to the input stream and expansion takes place normally (except in an \edef, where the expansion will take place when the \edefed macro will be used). –  egreg Apr 5 '13 at 22:07
    
@kahen Ahh, yes. He doesn't say, that the string must be enclosed in figure brackets to be assigned to \toks register. Now I got it working. –  user4035 Apr 5 '13 at 22:21

1 Answer 1

up vote 14 down vote accepted

A token register is very similar to a macro with no arguments, but there are differences in syntax and expansion rules.

Using the plain and latex definition of \toks@ for \toks 0 you can store the tokens abc via

\toks@{abc}

or

\def\tmp{abc}

Note that a macro has a specific command to set the macro (\def or wrappers around that such as \newcommand) but token registers just use the register name followed by a _balanced text group.

Conversely unlike macros which expand just by referencing their name, token registers, like other registers are expandable and their contnts are accessed by expanding \the.

So

\the\toks@

and

\tmp

each expand to abc.

Apart from these syntactic differences the expansion of token registers differs from macros in two important ways.

1) expansion in \edef and other expansion only contexts is limited to one level/

\def\tmp{aaa}

\def\tmpb{xx \tmp}

\toks@{xx \tmp}

\edef\A{\tmpb}

\edef\B{\the\toks@}

Now \tmpb and \toks@ contain the same token list xx \tmp but \A is defined by fully expanding expandable tokens and so has defnition xx aaa but the tokens returned by the are not further expanded in the edef and so \B has definition xx \tmp

2) The second difference is that as token registers do not have arguments # does not need to be (and is not) special when defining or expanding the register.

\toks{#1}

is just a token register consisting of a list of two tokens # and 1.

This is put to use in LaTeX's \g@addto@macro macro.

\g@addto@macro@\foo{abc}

is supposed to add abc to the end of the current definition of \foo.

A simple (and if I recall correctly original) definition could be

\makeatletter

\gdef\foo{123}

\def\gaA#1#2{%
  \expandafter\gdef\expandafter#1\expandafter{#1#2}}

\gaA\foo{abc}

\show\foo

That works fine and shows the definition as 123abc however try

\gaA\foo{#}

and you get

! Illegal parameter number in definition of \foo.

However we can use the fact that # isn't special in a toks register and that the register contents are only expanded once

\long\def\g@addto@macro#1#2{%
  \begingroup
    \toks@\expandafter{#1#2}%
    \xdef#1{\the\toks@}%
  \endgroup}

First the \toks@ register is defined to contain the expansion of the macro passed in the first argument, followed by the contents of the second argument. # is safe to use in a toks assignment. Then the macro is globally defined to be the expansion of \thetoks which is exactly the contents of \toks@ with no further expansion, even if that conatins # tokens:

\makeatletter

\gdef\foo{123}

\g@addto@macro\foo{abc}

\show\foo

\g@addto@macro\foo{#}

\show\foo

produces

> \foo=macro:
->123abc.
l.7 \show\foo

? 
> \foo=macro:
->123abc##.
l.11 \show\foo

where # has been added as intended (only one # has been added the doubled ## is an artefact of using \show.)

share|improve this answer
2  
The fact that # is not special makes toks registers particularly valuable for saving "metacode", i.e. code that is involved in the generation of more code, rather than execution. Macro definitions, parts of macro definitions, and so on, as opposed to things that need to be expanded as-is. By contrast, the expansion-inhibiting property doesn't seem to be as big a deal after eTeX's \unexpanded. (This is addressed more to the audience, of course.) –  Ryan Reich Apr 6 '13 at 0:03
    
What does the @ symbol after the register name mean? It is not mentioned in the book. I googled, but in vain. –  user4035 Apr 6 '13 at 8:46
2  
@user4035 -- the @ symbol in a command name means that this is an "internal" command, and to use it, it must either be within the scope of \makeatletter ... \makeatother, or in a .sty file where it is by default assumed to be of type "letter". –  barbara beeton Apr 6 '13 at 15:08
2  
@user4035 it's not after the name it is the command name toks@ (which is defined in latext o mean token register 0) –  David Carlisle Apr 6 '13 at 16:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.