# Could someone further elucidate expansion, catcodes, and scantokens…?

In response to my question "With TikZ is it possible to pass the node content through a preprocessor?", @MarkWibrow suggested a solution using \scantokens.

{\catcode\_=13 \gdef_{\rule[-1pt]{0.75em}{1.0pt}}}
\def\pp#1{{\catcode\_=13 \scantokens{#1\ignorespaces}}}


I've never understood why \scantokens would be useful or not, and I've seen plenty of warnings about the reputed dangers of \scantokens. The documentation in e-TeX is rather sparse (at least, too sparse to enlighten me at all). So, I decided to experiment a bit on my own and wrote my own version of \pp:

\def\aepp#1{{\catcode\_=13 #1}}


But

 \aepp{hello_world}


results in an error

! Missing $inserted. <inserted text>$


I thought, initially, that I perhaps didn't understand something about how \catcode works and tried the following MWE:

\documentclass{article}
{\catcode\_=13 \gdef_{\rule[-1pt]{0.75em}{1.0pt}}}
\setlength\parindent{0pt}
\begin{document}
{\catcode\_=13 hello_world}
\end{document}


which compiles without error.

I find this a bit confusing, because isn't

{\catcode\_=13 hello_world}


what \aepp{hello_world} expands to?

So, while I'm quite a bit in the dark about \scantokens, I'm more curious about why expansion of \aepp{hello_world} failed to accomplish what \pp{hello_world} does.

TeX's scanner (eyes) convert characters in a file to tokens. That only happens once, macro replacement text and all expansion processing processes tokens (which are [character-code,catcode] pairs.

the catcode table affects the conversion of characters to character tokens.

So your definition locally makes the catcode of _ 13 so if a _ character is encounted in that scope it would tokenize as a (_,13) token. But you pass it a token list as #1 of tokens that have already been tokenized so the catcode table is never consulted, so your change to the table is not used.

\scantokens acts as if the tokens are written to a file producing a stream of characters which are then read back so retokenised using the current catcode table.

Without \scantokens the classic construct you are looking for is:

\documentclass{article}
{\catcode\_=13 \gdef_{\rule[-1pt]{0.75em}{1.0pt}}}
\setlength\parindent{0pt}
\begin{document}
{\catcode\_=13 hello_world}

\def\aepp{\bgroup\catcode\_=13 \xaepp}
\def\xaepp#1{#1\egroup}
\aepp{hello_world}

\end{document}


which changes the catcode of _ before the argument is read. But this (like \verb not coincidentally) does not work if nested in the argument of another command, as then again the argument has already been tokenised.

• Thank you for the example of \xaepp: it made me think about this a bit more to understand how it worked around scanning tokens too early. – A.Ellett Nov 24 '14 at 0:22
• Will the future bring us a \verb that works inside arguments (by “actively changing catcodes” with \scantokes). Or is there a problem? – Manuel Nov 24 '14 at 0:23
• @Manuel there's always a problem:-) \verb| | is supposed to make three spaces but \scantokens{ } only makes one, unless you have already changed the catcode of space but if you've done that you don't need scantokens. (There are other problems but handling various white space issues are probably the hardest) – David Carlisle Nov 24 '14 at 0:34
• @DavidCarlisle I guess, since it's inline code, that spaces don't count too much. That's not a big deal. – Manuel Nov 24 '14 at 0:42
• @Manuel well it's easy for you to say that as you wouldn't be on the receiving end of the email bug reports if \verb became \verb-apart-from-space-and-#-and-%-and-{and-} :-) (which by the way is more or less \verb in tabularx which uses \meaning rather than \scantokens but has similar behaviour) – David Carlisle Nov 24 '14 at 0:50

The \scantokens command is not “dangerous” by itself, but it can have surprising effects.

How does it work? Its argument is scanned like for a \write operation, but all symbolic tokens are considered unexpandable, based on the current category codes; the result is placed in a “pseudofile” that is read in exactly as if \input was used.

This has various consequences; for instance an implicit space token is added at the end, actually an end-of-line character, exactly like TeX does with \input. This space can be neutralized in various ways, but the most common one is adding \empty or \noexpand at the end of the token list for \scantokens. Using \ignorespaces is not in my list of recommendations:

\scantokens{a\ignorespaces}X\scantokens{a\empty}X

\scantokens{a\ignorespaces} X\scantokens{a\empty} X


will output

so you see the effect of \ignorespaces goes beyond \scantokens.

One might be tempted to use \scantokens for getting verbatim in the argument to a command, but there's a catch:

\def\test#1{\begingroup\tt\catcode\\=12
\scantokens{#1}\endgroup}

\test{\abc\def}


will output

(I'm using Plain TeX for simplicity). Where does the space come from? In the internal representation for \write, control words are followed by a space; the change in category code for the backslash happens too late.

\def\test#1{\begingroup\tt\catcode\\=12
\scantokens{#1}\endgroup}
\newlinechar^^J

\test{a^^Jb^^J^^Jc}


will make TeX read the pseudofile as if it were

a
b

c


but the result is not

a b\par c

as it could be expected; rather, it is

because TeX converts the two consecutive end-of-lines into the token \par, not into the control sequence \par. This is proved by

\def\test#1{\begingroup\tt\catcode\\=12 \edef\par{\string\par}%
\scantokens{#1}\endgroup}
\newlinechar^^J

\test{a^^Jb^^J^^Jc}


that outputs

Exercise: explain the details.

Final quirk: one can use \scantokens\expandafter{\foo} and \foo will be expanded before \scantokens does its job:

\def\test#1{\begingroup\tt\catcode\\=12
\scantokens\expandafter{#1}\endgroup}

\def\foo{\abc\def}

\test\foo

\bye


will do the same as before

In order to have \scantokens inside \edef, one needs one more trick: the end-of-file must be hidden.

\def\escantokens#1#2#3{%
\begingroup\everyeof{\noexpand}%
#3%
\edef\x{\endgroup\def\noexpand#1{\scantokens{#2}}}\x
}

\escantokens\demo{\abc\def}{\catcode\\=12 }\show\demo

\escantokens\demo{\abc}{\def\abc{ABC}}\show\demo

\bye


The third argument to \escantokens is a set of temporary assignments (category codes, for instance, but not only).

The output on the terminal is

> \demo=macro:
->\abc \def .
l.8 \show\demo

?
> \demo=macro:
->ABC.
l.10 ...tokens\demo{\abc}{\def\abc{ABC}}\show\demo

?

• Regarding \ignorespaces, I noticed that approach posted in several places on this site, but when I tried \edef\demo{\scantokens{foo\ignorespaces}} I got an error of File ended while scanning definition of \demo. What's going on here? – A.Ellett Nov 24 '14 at 13:36
• @A.Ellett \scantokens inside \edef opens a new can of worms! – egreg Nov 24 '14 at 14:11
• @A.Ellett I added it – egreg Nov 24 '14 at 14:27