When does TeX execute the \catcode=`\... assignment?

Question

TeX always scans the input and converts it into some tokens, then expands some tokens according to the latest definition of that command ... stomach -> lists -> page

but when does TeX execute the catcode assignment? does it execute it as soon as looking at it?

for example

1. \catcode`\%=12
2. %
3. \catcode`\%=14
4. \def\foo{\catcode`\%=12%}
5. \bye

this will be wrong since TeX doesn't expand the macros to be defined. the last % in line 4 is only a comment sign, not a percent sign with catcode 12. But, does TeX execute line 1 and line 3 as soon as looking at (does it mean that TeX doesn't convert line 1 and line 3 into tokens?)

another example (I try to finish this macro, but failed)

\def\foo{hello, world}
{\catcode`\^=13
 \gdef^{\relax\ifmmode\def\next{\catcode`\^=7 ^}\else\let\next\foo\fi\next}}

{\catcode`\^=13
 ^$a^2$}
\bye

I think I should get the hello, world with a square just after it, because the first ^ should be expanded

^ ->\relax\ifmmode\def\next{\catcode`\^=7 ^}\else\let\next\foo\fi\next
  ->\relax\let\next\foo\next
  ->\relax\next
  ->\relax h e l l o , w o r l d

so, I should get hello, world. the second ^ in $ ... $ pairs

$a^2$ ->$a\relax\ifmmode\def\next{\catcode`\^=7 ^}\else\let\next\foo\fi\next2$
      ->$a\relax\def\next{\catcode`\^=7 ^}\next2$
      ->$a\relax\catcode`\^=7 ^2$
      ->$a^2$ % the catcode of ^ is 7 now

so, I think I should get a square

but this is wrong, I get mistake. I don't know why :-(

Answer 1

A \catcode`@=11 command is executed as soon as TeX encounters it when consuming tokens (as opposed to doing expansion).

Basically TeX works on a list of tokens trying to consume it. When it finds an expandable token it performs the expansion and continues to do so until an unexpandable token is found. In this case it sends it (with its arguments, which might require doing some expansion) to the "stomach" for execution.

Let's do some examples.

1. \catcode`@=11
   Since \catcode is unexpandable, TeX sends it with its arguments to the stomach and performs the assignment. All @ characters absorbed from this point on will have category code 11.

2. \def\at{`@}\catcode\at=11
   The macro definition is sent to the stomach; then \catcode is unexpandable and TeX determines its argument; since \catcode needs to see a number, \at is expanded and eventually \catcode`@=11 will be sent to the stomach and executed as before.

3. \def\foo{\catcode`@=11 }
   Here the \def token is unexpandable, so everything up to the closing brace is sent to the stomach; the \catcode assignment is not performed, because it's in the replacement text of a macro.

In the LaTeX kernel we find

448 \def\obeyspaces{\catcode`\ \active}
449 {\obeyspaces\global\let =\space}

(the line numbers are for finding the code). What does it mean? LaTeX needs to have available a definition for the active space, because this token (32,13) might sneak in some delayed write. In line 448 the \obeyspaces macro is defined to activate the space. In line 449 a group is opened, \obeyspaces is issued, which makes the space active, and the active space receives a definition. The end of the group will undo the category code assignment. Note that even if the definition of \space is

\def\space{ }

when TeX will expand the active space it will in turn expand \space producing a normal category code 10 space, because the replacement text for \space has been absorbed when the space has that catcode.

---

Let's now see your examples.

1  \catcode`\%=12
2  %
3  \catcode`\%=14
4  \def\foo{\catcode`\%=12%}
5  \bye

The assignment in line 1 is performed immediately, so the % in line 2 is simply printed. Then the assignment in line 3 is performed and TeX goes to line 4. The second % has category code 14, so it's a comment and the definition is not complete.

1  \def\foo{hello, world}
2  {\catcode`\^=13
3   \gdef^{\relax\ifmmode\def\next{\catcode`\^=7 ^}\else\let\next\foo\fi\next}}
4
5  {\catcode`\^=13
6   ^$a^2$}
7  \bye

You're trying to define the active ^. The group opened in line 2 will revert the category code assignment. However the ^ after \catcode`\^=7 has category code 13, not 7, because, as seen before, \catcode instructions in the replacement text of a macro being defined are not executed.

You just need to have available a category code 7 circumflex, as in the \obeyspace example above:

\def\foo{hello, world}
\def\normalhat{^}
{\catcode`\^=13
 \gdef^{\relax\ifmmode\let\next\normalhat\else\let\next\foo\fi\next}}

{\catcode`\^=13
 ^ $a^2$}
\bye

The output will be

hello, world a²