Could someone further elucidate expansion, catcodes, and scantokens...?

Question

In response to my question "With TikZ is it possible to pass the node content through a preprocessor?", @MarkWibrow suggested a solution using \scantokens.

{\catcode`\_=13 \gdef_{\rule[-1pt]{0.75em}{1.0pt}}}
\def\pp#1{{\catcode`\_=13 \scantokens{#1\ignorespaces}}}

I've never understood why \scantokens would be useful or not, and I've seen plenty of warnings about the reputed dangers of \scantokens. The documentation in e-TeX is rather sparse (at least, too sparse to enlighten me at all). So, I decided to experiment a bit on my own and wrote my own version of \pp:

\def\aepp#1{{\catcode`\_=13 #1}}

But

 \aepp{hello_world}

results in an error

! Missing $ inserted.
<inserted text> 
                $

I thought, initially, that I perhaps didn't understand something about how \catcode works and tried the following MWE:

\documentclass{article}
{\catcode`\_=13 \gdef_{\rule[-1pt]{0.75em}{1.0pt}}}
\setlength\parindent{0pt}
\begin{document}
{\catcode`\_=13 hello_world}
\end{document}

which compiles without error.

I find this a bit confusing, because isn't

{\catcode`\_=13 hello_world}

what \aepp{hello_world} expands to?

So, while I'm quite a bit in the dark about \scantokens, I'm more curious about why expansion of \aepp{hello_world} failed to accomplish what \pp{hello_world} does.

Answer 1

TeX's scanner (eyes) convert characters in a file to tokens. That only happens once, macro replacement text and all expansion processing processes tokens (which are [character-code,catcode] pairs.

the catcode table affects the conversion of characters to character tokens.

So your definition locally makes the catcode of _ 13 so if a _ character is encounted in that scope it would tokenize as a (_,13) token. But you pass it a token list as #1 of tokens that have already been tokenized so the catcode table is never consulted, so your change to the table is not used.

\scantokens acts as if the tokens are written to a file producing a stream of characters which are then read back so retokenised using the current catcode table.

Without \scantokens the classic construct you are looking for is:

\documentclass{article}
{\catcode`\_=13 \gdef_{\rule[-1pt]{0.75em}{1.0pt}}}
\setlength\parindent{0pt}
\begin{document}
{\catcode`\_=13 hello_world}

\def\aepp{\bgroup\catcode`\_=13 \xaepp}
\def\xaepp#1{#1\egroup}
\aepp{hello_world}


\end{document}

which changes the catcode of _ before the argument is read. But this (like \verb not coincidentally) does not work if nested in the argument of another command, as then again the argument has already been tokenised.

Answer 2

The \scantokens command is not “dangerous” by itself, but it can have surprising effects.

How does it work? Its argument is scanned like for a \write operation, but all symbolic tokens are considered unexpandable, based on the current category codes; the result is placed in a “pseudofile” that is read in exactly as if \input was used.

This has various consequences; for instance an implicit space token is added at the end, actually an end-of-line character, exactly like TeX does with \input. This space can be neutralized in various ways, but the most common one is adding \empty or \noexpand at the end of the token list for \scantokens. Using \ignorespaces is not in my list of recommendations:

\scantokens{a\ignorespaces}X\scantokens{a\empty}X

\scantokens{a\ignorespaces} X\scantokens{a\empty} X

will output

so you see the effect of \ignorespaces goes beyond \scantokens.

One might be tempted to use \scantokens for getting verbatim in the argument to a command, but there's a catch:

\def\test#1{\begingroup\tt\catcode`\\=12
  \scantokens{#1}\endgroup}

\test{\abc\def}

will output

(I'm using Plain TeX for simplicity). Where does the space come from? In the internal representation for \write, control words are followed by a space; the change in category code for the backslash happens too late.

Another (admittedtly tricky) quirk:

\def\test#1{\begingroup\tt\catcode`\\=12
  \scantokens{#1}\endgroup}
\newlinechar`^^J

\test{a^^Jb^^J^^Jc}

will make TeX read the pseudofile as if it were

a
b

c

but the result is not

a b\par c

as it could be expected; rather, it is

because TeX converts the two consecutive end-of-lines into the token \par, not into the control sequence \par. This is proved by

\def\test#1{\begingroup\tt\catcode`\\=12 \edef\par{\string\par}%
  \scantokens{#1}\endgroup}
\newlinechar`^^J

\test{a^^Jb^^J^^Jc}

that outputs

Exercise: explain the details.

Final quirk: one can use \scantokens\expandafter{\foo} and \foo will be expanded before \scantokens does its job:

\def\test#1{\begingroup\tt\catcode`\\=12
  \scantokens\expandafter{#1}\endgroup}

\def\foo{\abc\def}

\test\foo

\bye

will do the same as before

In order to have \scantokens inside \edef, one needs one more trick: the end-of-file must be hidden.

\def\escantokens#1#2#3{%
  \begingroup\everyeof{\noexpand}%
  #3%
  \edef\x{\endgroup\def\noexpand#1{\scantokens{#2}}}\x
}

\escantokens\demo{\abc\def}{\catcode`\\=12 }\show\demo

\escantokens\demo{\abc}{\def\abc{ABC}}\show\demo

\bye

The third argument to \escantokens is a set of temporary assignments (category codes, for instance, but not only).

The output on the terminal is

> \demo=macro:
->\abc \def .
l.8 \show\demo

? 
> \demo=macro:
->ABC.
l.10 ...tokens\demo{\abc}{\def\abc{ABC}}\show\demo

?