When does TeX execute the \catcode=\… assignment?

TeX always scans the input and converts it into some tokens, then expands some tokens according to the latest definition of that command ... stomach -> lists -> page

but when does TeX execute the catcode assignment? does it execute it as soon as looking at it?

for example

1. \catcode\%=12
2. %
3. \catcode\%=14
4. \def\foo{\catcode\%=12%}
5. \bye


this will be wrong since TeX doesn't expand the macros to be defined. the last % in line 4 is only a comment sign, not a percent sign with catcode 12. But, does TeX execute line 1 and line 3 as soon as looking at (does it mean that TeX doesn't convert line 1 and line 3 into tokens?)

another example (I try to finish this macro, but failed)

\def\foo{hello, world}
{\catcode\^=13
\gdef^{\relax\ifmmode\def\next{\catcode\^=7 ^}\else\let\next\foo\fi\next}}

{\catcode\^=13
^$a^2$}
\bye


I think I should get the hello, world with a square just after it, because the first ^ should be expanded

^ ->\relax\ifmmode\def\next{\catcode\^=7 ^}\else\let\next\foo\fi\next
->\relax\let\next\foo\next
->\relax\next
->\relax h e l l o , w o r l d


so, I should get hello, world. the second ^ in $...$ pairs

$a^2$ ->$a\relax\ifmmode\def\next{\catcode\^=7 ^}\else\let\next\foo\fi\next2$
->$a\relax\def\next{\catcode\^=7 ^}\next2$
->$a\relax\catcode\^=7 ^2$
->$a^2$ % the catcode of ^ is 7 now


so, I think I should get a square

but this is wrong, I get mistake. I don't know why :-(

• Do you mean when does TeX execute \catcode ... or when does it assign a catcode to a token? – Joseph Wright Apr 25 '13 at 10:29
• oh, sorry, I should give my example. – Lucas Li Apr 25 '13 at 10:32

A \catcode@=11 command is executed as soon as TeX encounters it when consuming tokens (as opposed to doing expansion).

Basically TeX works on a list of tokens trying to consume it. When it finds an expandable token it performs the expansion and continues to do so until an unexpandable token is found. In this case it sends it (with its arguments, which might require doing some expansion) to the "stomach" for execution.

Let's do some examples.

1. \catcode@=11
Since \catcode is unexpandable, TeX sends it with its arguments to the stomach and performs the assignment. All @ characters absorbed from this point on will have category code 11.

2. \def\at{@}\catcode\at=11
The macro definition is sent to the stomach; then \catcode is unexpandable and TeX determines its argument; since \catcode needs to see a number, \at is expanded and eventually \catcode@=11 will be sent to the stomach and executed as before.

3. \def\foo{\catcode@=11 }
Here the \def token is unexpandable, so everything up to the closing brace is sent to the stomach; the \catcode assignment is not performed, because it's in the replacement text of a macro.

In the LaTeX kernel we find

448 \def\obeyspaces{\catcode\ \active}
449 {\obeyspaces\global\let =\space}


(the line numbers are for finding the code). What does it mean? LaTeX needs to have available a definition for the active space, because this token (32,13) might sneak in some delayed write. In line 448 the \obeyspaces macro is defined to activate the space. In line 449 a group is opened, \obeyspaces is issued, which makes the space active, and the active space receives a definition. The end of the group will undo the category code assignment. Note that even if the definition of \space is

\def\space{ }


when TeX will expand the active space it will in turn expand \space producing a normal category code 10 space, because the replacement text for \space has been absorbed when the space has that catcode.

1  \catcode\%=12
2  %
3  \catcode\%=14
4  \def\foo{\catcode\%=12%}
5  \bye


The assignment in line 1 is performed immediately, so the % in line 2 is simply printed. Then the assignment in line 3 is performed and TeX goes to line 4. The second % has category code 14, so it's a comment and the definition is not complete.

1  \def\foo{hello, world}
2  {\catcode\^=13
3   \gdef^{\relax\ifmmode\def\next{\catcode\^=7 ^}\else\let\next\foo\fi\next}}
4
5  {\catcode\^=13
6   ^$a^2$}
7  \bye


You're trying to define the active ^. The group opened in line 2 will revert the category code assignment. However the ^ after \catcode\^=7 has category code 13, not 7, because, as seen before, \catcode instructions in the replacement text of a macro being defined are not executed.

You just need to have available a category code 7 circumflex, as in the \obeyspace example above:

\def\foo{hello, world}
\def\normalhat{^}
{\catcode\^=13
\gdef^{\relax\ifmmode\let\next\normalhat\else\let\next\foo\fi\next}}

{\catcode\^=13
^ $a^2$}
\bye


The output will be

hello, world a2

• Thanks, egreg. When reading The TeXbook, Knuth said:When TeX encounters a control sequence in the token list, it look up the current meaning, and in certain cases it will expand that token into a sequence of other tokens before continuing to read, I am curious about this: how does TeX know the current meaning of a token. now from item 3 of your list, I understand the why behind the scenes. – Lucas Li Apr 25 '13 at 11:14
• Let me have a think. I still have some questions: 1. I write the TeX code into a file, does TeX read my source file and convert my input into tokens (at this time, when encountering \gdef, it only converts it into a gdef token rather that send it into the stomach. Am I right?). We call this the first pass. 2. in the first pass, how does TeX process the replacement texts of a macro? What catcodes are assigned to the characters in the replacement texts? 3. After the first pass, TeX have converted everything in the source file into tokens. Now it begins to expand these tokens, (continue) – Lucas Li Apr 25 '13 at 11:49
• does TeX send the unexpandable tokens to the stomach when looking at this token, for instance, gdef,etc. at this time(the second pass)? (sorry, sir, I am still a stupid guy. In order to understand why, I have to ask you these questions) – Lucas Li Apr 25 '13 at 11:53
• @TimLi Answering your questions in these comments requires a chapter in a book. You can find it very cleanly explained in V. Eijkhout's "TeX by Topic", which is available on CTAN (and probably in your TeX distribution). – egreg Apr 25 '13 at 12:12
• many thanks, many thanks. I will read this book. as you said to me before, "texdoc is your friend." :-) – Lucas Li Apr 25 '13 at 12:23