This is a "How does TeX know when to stop" question.
Case 1
Note: I have since found out that my understanding here is wrong. See SPOILER and answers.
When TeX is constructing a macro name, it consumes all category 11 tokens until it scans a non-category 11 token. The non-category 11 token is
- Scanned: TRUE
- Typeset: FALSE
Case 2
When TeX is constructing a register value, it consumes all tokens of category 12 (number) until it finds a non-category 12 token. The non-category 12 token is somehow both
- Scanned: TRUE
- Typeset: TRUE
Code
\documentclass{article}
\usepackage{fontspec}% xelatex
\begin{document}
\section{Macro Name Scanning}
\def\macro{123}
\count1=\macro
4
This is a register: \the\count1. Notice that the first 4 was consumed by the TeX scanner as expected.
\section{Register Value Scanning}
\count1=42
\advance\count1 by1000The < The \textit{T} in \textit{The} should have been consumed by the TeX scanner before being typeset. Where does \textbackslash relax come into play here?% Why is the T not consumed by the number scanner (catcode 12)
\end{document}
Output (typeset)
Notes
I expected that I would need a \relax
before the T in case 2 as I do after \macro
in case 1.
SPOILER ALERT
My confusion is explained in the answers below, but I'd like to point out something I learned:
A difference between macro construction (scanning csnames) and register value construction (scanning number values) is:
- control sequence name/macro name scanning: TeX scans characters bytes (tokens not yet created, rather being actively created at this step)
- register value scanning: TeX scans character tokens (tokenization has already happened. A character token is an integer pair consisting of (character code, cat code))
This has important implications.
- control sequence name/macro name scanning sees all character bytes of a file and applies certain TeX rules of tokenization (e.g. when to leave out certain bytes like spaces or newlines after macros)
- register value scanning never sees character bytes, rather the resulting character token stream after bytes have been processed. A macro token is expanded when encountered and newline character bytes have already been converted to catcode 10 (i am not sure where catcode 5 comes into play if at all here). There is a space rule at this stage too. Catcode 10 tokens are removed when read, and not put back into the input stream.
P.S. I sure hope I have the details right!
The\macro story goes...
andThe\count1=123 story goes...
. Both spaces (after \macro and \count1=123) were missing in the resulting PDF. Maybe it has something to do with the new line?\count1=\macro 4
and look at the value of\count1
. Then try\count1=\macro\space 4
.x\count1=\macro\space 4
and you'll see no space between x and 4 in the printout. Note that\space
is not “backslash space”.x\count1=\macro\space 4
is seven tokens. The control sequence\space
is tokenized.