5

I read some macros of manmac.tex, when I read this code snippet

% (now Appendix E resumes again)
% macros for verbatim scanning
\chardef\other=12
\def\ttverbatim{\begingroup
  \catcode`\\=\other
  \catcode`\{=\other
  \catcode`\}=\other
  \catcode`\$=\other
  \catcode`\&=\other
  \catcode`\#=\other
  \catcode`\%=\other
  \catcode`\~=\other
  \catcode`\_=\other
  \catcode`\^=\other
  \obeyspaces \obeylines \tt}

\outer\def\begintt{$$\let\par=\endgraf \ttverbatim \parskip=\z@
  \catcode`\|=0 \rightskip-5pc \ttfinish}
{\catcode`\|=0 |catcode`|\=\other % | is temporary escape character
  |obeylines % end of line is active
  |gdef|ttfinish#1^^M#2\endtt{#1|vbox{#2}|endgroup$$}}

In definition of \begintt, the catcode of \ is changed to 12 by \ttverbatim, then comes a sequence of macros \parskip=\z@ \catcode`\|=0 \rightskip-5pc \ttfinish. In order to describe my question clearly, I extracted the author's idea in the following code snippet:

\def\x{x}
\def\myverbatim{\begingroup\catcode`\\=12 \tt \x }
{\catcode`\^^@=0 ^^@myverbatim ^^@endgroup}

this output is x. What makes me feel confused is that, since the catcode of \ is changed from 0 to 12, why the following input text \tt \x can still be analyzed into control sequence tokens? The output that I expect is \tt \x.

2
  • Do you have TeXbook? Then maybe read the notice right after \def\listing part
    – user202729
    May 15 at 11:45
  • @user202729 yeah, I have. \listing in Appendix D: Dirty Tricks. Let me read it now. But is that part really relevant to this question?
    – Lucas
    May 15 at 11:51

2 Answers 2

7

Because when you defined \def\myverbatim{\begingroup\catcode`\\=12 \tt \x }, the catcode of \ was 0, so TeX already tokenized \tt and \x as control sequence tokens, and TeX never(ish) changes a token after it has already been tokenized. Maybe this example helps you see more clearly the difference:

\def\x{x}
\def\myverbatim{\begingroup\catcode`\\=12 (1: \tt \x)}
\def\endmyverbatim{\endgroup}
\catcode`|=0

\myverbatim % (1: \tt \x)

(2:\tt \x)
|endmyverbatim

\bye

In the case of 1, TeX has already scanned \tt \x as control sequence tokens, and when you expand \myverbatim, they remain being control sequence tokens and are processed accordingly.

After fully processing \myverbatim, the catcode of \ is changed, and then TeX proceeds to scan (2:.... At this point, TeX sees a \ as an ordinary character, so \tt \x is just a string of 6 characters with no special meaning. So much so that you now have to type |endmyverbatim to get the environment to end correctly (\endmyverbatim would just be typeset).


This process is what Knuth calls TeX's eyes and TeX's mouth. The eyes look at the stream of characters and assign each one a meaning based on the current catcode settings. This meaning is fixed to each character seen, and the character + meaning makes a token. The stream of tokens then reaches the mouth, where they are processed according to their meaning.

This tokens-don't-change thing is what makes \makeatletter...\makeatother not work inside a definition, and what makes verbatim environments not work when used in the argument to another macro.

4
  • Thanks. I don't understand the last paragraph you said, understand what you said in other paragraphs :) After read your answer, I know where I made mistakes. I thoguth tex doesn't analyze the replacement text in a macro into tokens, but the opposite is true. In terms of my feeling, it is not easy to design a tex macro(especially the macros for an environment), I have to be always careful how tex parse my codes. thanks again.
    – Lucas
    May 15 at 12:35
  • from your words, I guess that tex expands a non-primitive control sequence token once it finished converting a control sequence into a control sequence token.
    – Lucas
    May 15 at 12:58
  • @Lucas The note in the last paragraph is just a consequence of what's in the answer. People do \def\xx{\makeatletter \some@macro \makeatother} expecting that to work as \makeatletter\def\xx{\some@macro}\makeatother, but it doesn't because \some@macro is scanned as the control sequence \some and the characters @macro while @ has catcode 12. For the same reason, \def\xx#1{#1} \xx{\verb|$p&ci#l|} doesn't work, because the argument of \xx is scanned and tokenized, and only then passed to \verb, but then it's too late because $ is already a math-shift token and so on. May 15 at 13:06
  • @Lucas On your second comment: yes. The process is more or less: 1) TeX sees a ``; 2) TeX scans ahead for either a) a sequence of characters with catcode 11 or b) a single character with catcode 12 (these two steps were the eyes), then; 3) TeX expands the scanned control sequence into its meaning (the mouth) and finally; 4) processes the tokens inserted by expanding that control sequence. May 15 at 13:09
3

When TeX is absorbing the replacement text for a macro to be defined with \def or \gdef it does no interpretation whatsoever to the input, it only tokenizes it according to the category codes valid when \def was found. With \edef or \xdef only macro expansion is performed.

In either case, no assignment is performed, so it would be useless to try \edef\myverbatim.

With your definition, the replacement text of \myverbatim is

|begingroup| |catcode| `12 |\| =12 112 212 |tt| |x|

where |...| denotes a symbolic token (spaces are just for clarity).

Doing \edef would only produce a weird error, because the only expandable token is \\, which in plain TeX is

\\:
macro:#1pt->#1

(for historical reasons).

Only after you call \myverbatim the replacement text is inserted in the input stream, tokens are expanded, assignments are performed and thus the category code of \ will become 12.

Why is this so? A typical assignment is \advance\mycount by 1 and you want that TeX does this at call time, not when the macro that will eventually does it is defined. Similarly, if you have a macro in the replacement text you want that its meaning at call time is used, not the one at definition time.

Similarly, you want that the category codes valid at definition time are the same as those at call time, or very weird effects would ensue. For instance, the definition of \begintt has $$ and you want that these are always $3, not whatever could be at call time.

2
  • after reading your answer, sir, macros only encapsulate the computation not do the computation.
    – Lucas
    May 15 at 13:30
  • @Lucas That's a way to think to this business
    – egreg
    May 15 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.