After changing the catcode of the escape character `\`, why can this code still be compiled successfully?

Question

I read some macros of manmac.tex, when I read this code snippet

% (now Appendix E resumes again)
% macros for verbatim scanning
\chardef\other=12
\def\ttverbatim{\begingroup
  \catcode`\\=\other
  \catcode`\{=\other
  \catcode`\}=\other
  \catcode`\$=\other
  \catcode`\&=\other
  \catcode`\#=\other
  \catcode`\%=\other
  \catcode`\~=\other
  \catcode`\_=\other
  \catcode`\^=\other
  \obeyspaces \obeylines \tt}

\outer\def\begintt{$$\let\par=\endgraf \ttverbatim \parskip=\z@
  \catcode`\|=0 \rightskip-5pc \ttfinish}
{\catcode`\|=0 |catcode`|\=\other % | is temporary escape character
  |obeylines % end of line is active
  |gdef|ttfinish#1^^M#2\endtt{#1|vbox{#2}|endgroup$$}}

In definition of \begintt, the catcode of \ is changed to 12 by \ttverbatim, then comes a sequence of macros \parskip=\z@ \catcode`\|=0 \rightskip-5pc \ttfinish. In order to describe my question clearly, I extracted the author's idea in the following code snippet:

\def\x{x}
\def\myverbatim{\begingroup\catcode`\\=12 \tt \x }
{\catcode`\^^@=0 ^^@myverbatim ^^@endgroup}

this output is x. What makes me feel confused is that, since the catcode of \ is changed from 0 to 12, why the following input text \tt \x can still be analyzed into control sequence tokens? The output that I expect is \tt \x.

Answer 1

Because when you defined \def\myverbatim{\begingroup\catcode`\\=12 \tt \x }, the catcode of \ was 0, so TeX already tokenized \tt and \x as control sequence tokens, and TeX never(ish) changes a token after it has already been tokenized. Maybe this example helps you see more clearly the difference:

\def\x{x}
\def\myverbatim{\begingroup\catcode`\\=12 (1: \tt \x)}
\def\endmyverbatim{\endgroup}
\catcode`|=0

\myverbatim % (1: \tt \x)

(2:\tt \x)
|endmyverbatim

\bye

In the case of 1, TeX has already scanned \tt \x as control sequence tokens, and when you expand \myverbatim, they remain being control sequence tokens and are processed accordingly.

After fully processing \myverbatim, the catcode of \ is changed, and then TeX proceeds to scan (2:.... At this point, TeX sees a \ as an ordinary character, so \tt \x is just a string of 6 characters with no special meaning. So much so that you now have to type |endmyverbatim to get the environment to end correctly (\endmyverbatim would just be typeset).

---

This process is what Knuth calls TeX's eyes and TeX's mouth. The eyes look at the stream of characters and assign each one a meaning based on the current catcode settings. This meaning is fixed to each character seen, and the character + meaning makes a token. The stream of tokens then reaches the mouth, where they are processed according to their meaning.

This tokens-don't-change thing is what makes \makeatletter...\makeatother not work inside a definition, and what makes verbatim environments not work when used in the argument to another macro.

Answer 2

When TeX is absorbing the replacement text for a macro to be defined with \def or \gdef it does no interpretation whatsoever to the input, it only tokenizes it according to the category codes valid when \def was found. With \edef or \xdef only macro expansion is performed.

In either case, no assignment is performed, so it would be useless to try \edef\myverbatim.

With your definition, the replacement text of \myverbatim is

|begingroup| |catcode| `12 |\| =12 112 212 |tt| |x|

where |...| denotes a symbolic token (spaces are just for clarity).

Doing \edef would only produce a weird error, because the only expandable token is \\, which in plain TeX is

\\:
macro:#1pt->#1

(for historical reasons).

Only after you call \myverbatim the replacement text is inserted in the input stream, tokens are expanded, assignments are performed and thus the category code of \ will become 12.

Why is this so? A typical assignment is \advance\mycount by 1 and you want that TeX does this at call time, not when the macro that will eventually does it is defined. Similarly, if you have a macro in the replacement text you want that its meaning at call time is used, not the one at definition time.

Similarly, you want that the category codes valid at definition time are the same as those at call time, or very weird effects would ensue. For instance, the definition of \begintt has $$ and you want that these are always $3, not whatever could be at call time.