How does TeX know whether to eat this space if its catcode is about to change?

Question

In the document

\catcode32=13\def {o} \bye

how does TeX know to tokenize the space after \def as an active character?

Details: The correct result is o. First, we set spaces (ASCII 32) to be active characters (category 13); then, we define them to print o; then, we add a space, printing o, and finish.

How does the parser recognize the end of the literal number 13? A literal number can't contain an unexpandable control sequence, like \def. But it can contain an expandable control sequence like \if, and we can't know which this is until we finish reading it and look it up. So, until we reach the end of parsing the \def token, we don't know where the end of 13 is.

It gets hairy here. Multi-character control sequences absorb the following space. So before the \catcode assignment ran, the \def would have absorbed the following space. But to run the \catcode assignment, we have to tokenize up to the end of the \def token.

So, how does TeX know not to eat that space?

Answer 1

It is true that if the token processor creates a token and it is sent to the expand processor, then the once assigned catcode cannot be changed (there are several exceptions like \string, \detokenize etc. but is out of our current focus).

But your example

\catcode32=13\def {o} \bye

doesn't meet the second part of the sentence: the space after \def is waiting for processing at the token-processor level. It is not sent to the expand processor. The token processor recognizes the space after \def twice. Firstly, when \def control sequence is constructed (space has catcode 10 in this time). Then the token processor switches to the "skipping spaces" state, it sends the \def to the expand processor, and it keeps the space character for further processing. Expand processor cannot expand \def, so it tells the main processor that the number reading is finished, and the main processor finalizes the \catcode assignment and runs \def. It deactivates expand processor at this place and asks to the token processor to move the next token. The token processor gets waiting for space and creates a token from it using current catcode 13. Its "skipping spaces" state is finished because the created token isn't space with category 10. It sends the token space/13 to the main processor.

The main lesson learned from it is: the token processor doesn't definitely create the pair character/catcode until it is sent to the next level of processing (expand processor, main processor).

Answer 2

\catcode32=13\def {o} \bye

1. While in the process of gathering tokens that form the second TeX-⟨number⟩-quantity of the \catcode-assignment, that TeX-⟨number⟩-quantity is found to be an ⟨integer constant⟩ consisting of a sequence of ⟨digit⟩-tokens with leading ⟨digit⟩-tokens 112 and 312.
2. As the ⟨digit⟩-token 312 is considered a component of the ⟨integer constant⟩ and thus not suitable for terminating the process of gathering tokens belonging to the ⟨integer constant⟩, more tokens are to be tokenized and expanded if expandable, until finding a token suitable for terminating the process of gathering tokens belonging to the ⟨integer constant⟩ as not considered a component of the ⟨integer constant⟩.
3. Thus, in the course of forming more tokens probably belonging to the second TeX-⟨number⟩-quantity of the catcode-assignment, TeX gathers characters denoting the next token.
4. Hereby TeX encounters the character \ which has category code 0 (escape) and realizes that characters denoting a control sequence token are to be gathered. After realizing this, TeX encounters the character d and due to the character d having category code 11 (letter) realizes that characters denoting a control word token are to be gathered.
5. So, TeX also gathers the character e and the character f which both have category code 11 (letter).
6. Then TeX encounters the space character, which at that time has category code 10 (space). Characters not of category code 11 (letter) at the time of tokenizing are not considered components of names of control word tokens. Thus the space character terminates the process of gathering characters denoting a control word token and TeX appends the control word token \def to the token stream and switches the reading-apparatus to state S (skipping blanks).

The token \def is not considered a component of an ⟨integer constant⟩, thus the process of gathering tokens belonging to the ⟨integer constant⟩/the process of gathering tokens belonging to the second TeX-⟨number⟩-quantity of the catcode-assignment is terminated and the catcode-assignment is performed.

Thus, when processing the space character, which terminated gathering characters for the \def-token, takes place, the category code of the space character is 13 (active). As state S (skipping blanks) only affects characters of category code 10 (space), the space character gets tokenized as an active character token.

If you like to get familiar with how in TeX the stage of tokenization/processing single characters of a line of .tex-input is intertwined with other stages of processing (stage of expansion / stage of performing assignments), try to figure out why

\catcode`\.=0\def.macro{replacement}\macro\bye

works out while

\catcode`\.=0.def.macro{replacement}\macro\bye

yields error-messages about .macro/\macro being undefined.

Answer 3

It's simple, actually.

1. The \catcode token alone tells TeX that a category code assignment is to be performed;

2. in this case, TeX looks for an 8-bit ⟨number⟩ (it's 21-bit for XeTeX or LuaTeX);

3. upon having found it, TeX looks for an ⟨optional equals⟩ (gobbling a space after =, if both are present);

4. TeX looks for a 4-bit number.

How the ⟨number⟩ can be specified is too long to describe. In your case, both the required ⟨number⟩s are explicit constants, and a category 10 space would be gobbled after the sequence of digits.

It's also important to remember that an entire line of input is read, and the end-of-record is replaced by the \endlinechar, after throwing out trailing spaces, but the characters haven't yet been tokenized: TeX tokenizes the input only when it needs to.

So the following code lines are completely equivalent:

\catcode32=13\def {o} \bye
\catcode32 =13\def {o} \bye
\catcode32= 13\def {o} \bye
\catcode32=13 \def {o} \bye
\catcode32 = 13\def {o} \bye
\catcode32 =13 \def {o} \bye
\catcode32 = 13 \def {o} \bye

Indeed, the input

{\catcode32=13\def {o} }\par
{\catcode32 =13\def {o} }\par
{\catcode32= 13\def {o} }\par
{\catcode32=13 \def {o} }\par
{\catcode32 = 13\def {o} }\par
{\catcode32 =13 \def {o} }\par
{\catcode32 = 13 \def {o} }\par
\bye

produces seven o's. Your case is the first one, where \def cannot be interpreted as a digit: the token is put aside to be reread after the assignment has been performed.

Let's examine

\catcode32=13 \def {o} \bye

that's what you probably want to know about (the other two with a space after 13 are similar). When TeX has scanned the =, it proceeds to look for a 4-bit ⟨number⟩. It finds 1, so it knows that an explicit constant follows. Next comes 3 and further comes a space. No category code assignment has yet been done, so this space is given category code 10 and terminates the lookup for digits. Now TeX know what category code to assign, it performs the assignment and discards the category 10 character that follows.

Maybe you can ask what would happen with

\catcode32=13  \def {o} \bye

Exactly the same, because after scanning a category code 10 character, TeX enters the state “skipping blanks”, so the following category 10 characters are ignored. This action of skipping blanks happens before the assignment of category code is performed, because TeX wants to “normalize” the input stream.

Exercise: predict the output of

\catcode32=13\def {o} \  \bye

(there are two spaces after the backslash).

Answer 4

From experimenting with things such as

\catcode`Z=13Z

and

\def\defZ{W}\catcode`Z=13\defZ{XY}

I feel that in trying to terminate the catcode assignment TeX scans ahead with current pre-existing catcode regimen. So it will see \def<space> decide that \def terminates the assignment, then resume its scanning and catcode assignments (unlike in a \futurelet which would have frozen the catcode) so the space is now active.

Not an authoritative answer because not based on actually going to the TeX sources...

---

The following example

  {\catcode`Z=13 \gdefZ{WWW}}\catcode`Z=13ZZ

which prints out ZWWW shows that at least here the catcode 11 Z (the first one after 13) is not reinterpreted as catcode13.

This must be added to my earlier babblings (the additional babble being that some catcode may get frozen; in OP, it is farther away though, but let's wait for the correct answers).