# Why does the TeX scanner process tokens for register numbers and macro names differently?

This is a "How does TeX know when to stop" question.

# Case 1

Note: I have since found out that my understanding here is wrong. See SPOILER and answers.

When TeX is constructing a macro name, it consumes all category 11 tokens until it scans a non-category 11 token. The non-category 11 token is

• Scanned: TRUE
• Typeset: FALSE

# Case 2

When TeX is constructing a register value, it consumes all tokens of category 12 (number) until it finds a non-category 12 token. The non-category 12 token is somehow both

• Scanned: TRUE
• Typeset: TRUE

# Code

\documentclass{article}
\usepackage{fontspec}% xelatex
\begin{document}
\section{Macro Name Scanning}
\def\macro{123}
\count1=\macro
4
This is a register: \the\count1. Notice that the first 4 was consumed by the TeX scanner as expected.

\section{Register Value Scanning}
\count1=42
\advance\count1 by1000The < The \textit{T} in \textit{The} should have been consumed by the TeX scanner before being typeset. Where does \textbackslash relax come into play here?% Why is the T not consumed by the number scanner (catcode 12)
\end{document}


# Notes

I expected that I would need a \relax before the T in case 2 as I do after \macro in case 1.

# SPOILER ALERT

My confusion is explained in the answers below, but I'd like to point out something I learned:

A difference between macro construction (scanning csnames) and register value construction (scanning number values) is:

• control sequence name/macro name scanning: TeX scans characters bytes (tokens not yet created, rather being actively created at this step)
• register value scanning: TeX scans character tokens (tokenization has already happened. A character token is an integer pair consisting of (character code, cat code))

This has important implications.

• control sequence name/macro name scanning sees all character bytes of a file and applies certain TeX rules of tokenization (e.g. when to leave out certain bytes like spaces or newlines after macros)
• register value scanning never sees character bytes, rather the resulting character token stream after bytes have been processed. A macro token is expanded when encountered and newline character bytes have already been converted to catcode 10 (i am not sure where catcode 5 comes into play if at all here). There is a space rule at this stage too. Catcode 10 tokens are removed when read, and not put back into the input stream.

P.S. I sure hope I have the details right!

• The last paragraph is quite misleading. – egreg Jan 19 '17 at 15:34
• @egreg How so? I tested The\macro story goes... and The\count1=123 story goes.... Both spaces (after \macro and \count1=123) were missing in the resulting PDF. Maybe it has something to do with the new line? – Jonathan Komar Jan 19 '17 at 15:39
• Try \count1=\macro 4 and look at the value of \count1. Then try \count1=\macro\space 4. – egreg Jan 19 '17 at 15:41
• There is no space: try x\count1=\macro\space 4 and you'll see no space between x and 4 in the printout. Note that \space is not “backslash space”. – egreg Jan 19 '17 at 16:00
• x\count1=\macro\space 4 is seven tokens. The control sequence \space is tokenized. – egreg Jan 19 '17 at 16:09

## 3 Answers

I don't think that your description of scanning for a macro name is accurate (although I do not know what you mean by typeset:false, so that may or may not be true)

Note that scanning for macro names happens before tokenisation (or rather during it) as it is the determinisation of whether to make a csname token. After the character (not token) that has catcode 0 is seen then if the next character has catcode 11 all further catcode 11 characters are scanned as part of the csname, the first non-catcode 11 character is consumed and not tokenised if it would make a token of catcode 10 (space) otherwise it is put back into the input stream.

Numbers are part of tex's inner execution (what the TeXBook calls the stomach) so happen after tokenisation and after macro expansion. If TeX is parsing for a <number> then it will expand all tokens until it finds an unexpandable token that is part of a number syntax (so catcode 11 or 12 digits, or after " also hex letters A-F and other forms such as alphabetic character code numbers such as a or \a The first non expandable token that is not part of the syntax for <number> is always returned (as a token) to the input stream unless it is a space token, which is your T here.

• @macmadness86 only very indirectly, obviously anything that is typeset must have been passed to tex's typesetting routines so must have been tokenised at some point, but other than that typesetting is in no way involved here. consider def\foo{\abc .} and \def\foo{\abc.} nothing is typeset (unless \foo gets used later in a typesetting context) but they are equivalent definitions as the space character is never tokenised due to the rules above so in either case the replacement text of \foo is the two tokens \abc and . – David Carlisle Jan 19 '17 at 10:28
• Thanks for answering. I was about to rephrase my first comment, because as I think about it, I understand more :) I should have left it. Bad timing. – Jonathan Komar Jan 19 '17 at 10:29
• +check because the last sentence was so enlightening: "The first non expandable token that is not part of the syntax for <number> is always returned (as a token) to the input stream unless it is a space token, which is your T here." Of course egreg's explanation of "radix" indicators is essential too. – Jonathan Komar Jan 19 '17 at 11:20
• @macmadness86 the important part is that you gave the tick to me not egreg, he's always so pleased for me when that happens:-) – David Carlisle Jan 19 '17 at 11:53
• You're probably right. It was the right thing to do--you, with a mere 366,000 points. In summary: An important difference between how TeX scans a csname vs how TeX scans a register value is that for csnames, TeX consumes a space and stops scanning. On the other hand, when TeX scans numbers for a register value, it scans spaces but keeps scanning for more numbers. The only letters that can be numbers are A,B,C,D,E,F when using them to represent hexidecimal numbers. I hope this is a correct assessment of the process. – Jonathan Komar Jan 19 '17 at 12:01

Your assumptions are wrong. When doing an assignment to a count register, TeX obeys the syntax

<count register><optional equal sign><number>


where <count register> can be an internal numeric register (for instance \hyphenpenalty), a \countdef token (after \newcount\foo or a direct \countdef\foo=1) or \count<number>.

The <optional equal sign> can be empty, a space token, or =12 surrounded by (optional) space tokens (well it's a bit more complicated, but not really relevant).

A <number> consists of an optional radix indicator

 ' "


and then a sequence of digits in the corresponding radix. For the case of a backquote, the “digit” is a single character or a control sequence of length one (alphabetic constant). For ' the digits are among 01234567 (with category code 12); for " the digits are 0123456789ABCDEF (where the letters can have category code 11 or 12, but 0123456789 must have category code 12). Without an indicator, decimal is assumed and the admissible digits are 0123456789 with category code 12.

The <number> ends when, after macro expansion, TeX finds a token that cannot be interpreted as a suitable digit for the chosen radix. If this token is a space, it is gobbled.

Your example

\def\macro{123}

\count1=\macro
4


assigns to \count1 the integer 1234, because there is nothing between \macro and 4, as spaces are ignored during tokenization after a control word (and the end-of-line character gets converted to a space during the same tokenization phase). If you want to assign 123 and print 4, you have to say

\count1=\macro\space 4


(whether 4 is on the next line is irrelevant).

The syntax for \advance is similar (instead of = there is the optional by); so in

\advance\count1 by 1000T


the search for digits will stop at T, but it would stop at ! as well.

Some more words on the assignments above. With

\count1=\macro
4


TeX sees six tokens

\count • 1 • = • \macro • 4 • ⍽


(spaces are just for clarity, • separates tokens from each other). Exactly the same would be found with

\count1=\macro4
\count1=\macro 4


because of the rules about tokenization. There is no space token after \macro. The final ⍽ denotes the space token deriving from the end-of-line, which is tokenized because it doesn't get ignored during macro name constructions.

Since \count is unexpandable, TeX “executes” it, which means it has to perform an assignment, so a <number> is searched for; the following token is 1 which is right; the next token is =, so the search for digits stops and TeX knows it has to assign a value to \count1. The = is gobbled because it optional and TeX starts searching for a number. The first token is expandable, so TeX expands it getting

1 • 2 • 3 • 4 • ⍽


Now ⍽ stops the search for digits and the assignment can be performed. This space token is gobbled by rule.

Let's examine

\count1=42
\advance\count1 by1000The


and divide it into tokens:

\count•1•=•4•2•⍽•\advance•\count•1•⍽•b•y•1•0•0•0•T•h•e


(no typographic space for compactness, • still separates tokens from each other). The assignment of 42 to \count1 is performed as before and the space token is gobbled. We remain with

\advance•\count•1•⍽•b•y•1•0•0•0•T•h•e


The execution of \advance makes TeX to search for an appropriate register name, which it finds as \count1 (the space token is ignored after the lookup for the register's number). Next by is gobbled because it is an optional keyword and we remain with

1•0•0•0•T•h•e


TeX is looking for a <number> so it does as before, with macro expansion. However there is no macro to expand and the search for digits stops at the first token that is not a digit for the radix 10, so at T. The assignment is performed and \count1 is assigned the value 1042. Next the token T is reexamined and it will start a paragraph.

Be careful because something like

\count1="1000At this point


will assign to \count1 the value 65546 and start a paragraph with t.

Always end constants with an unexpandable token that's not a digit. A space is good for this as it will be ignored.

• The hard part for me is reconciling "there is nothing between \macro and 4" (hence 4 is not typeset, rather assigned to the register--because TeX will gobble the 4 while constructing the csname) with "the search for digits will stop at T" (hence T is typeset and NOT assigned to the register). I think David hit the nail on the head when he wrote "consumed and not tokenised [...] if it would make a token of catcode 10 (space) otherwise it is put back into the input stream". Perhaps my knowledge gap is at the tokenization stage. – Jonathan Komar Jan 19 '17 at 10:21
• @macmadness86 Why should TeX gobble the 4? If you type \macro4 you get 1234 – egreg Jan 19 '17 at 10:30

What do you mean by "constructing a macro name"?

At the time of reading/tokenizing input it is not important whether the control sequence token in question will denote a macro or a primitive or an undefined control sequence.

At the time of reading and tokenizing input, there are no tokens yet, thus names of control sequence tokens do not consist of whatsoever tokens. They are formed by characters (not character tokens). \string can be used for obtaining a sequence of corresponding character tokens. \csname..\endcsname offers a means of denoting control sequence tokens by means of sequences of expandable tokens and character-tokens.

Always take care about when it is appropriate to use the term "token" and when it is not appropriate to use the term "token" as the -eh- "object of contemplation" is not a token/is not something that came into being as a result of tokenization or expansion.

Control sequence tokens come in two flavours:
1. Control symbol tokens.
2. Control word tokens.

A sequence of two characters in the input where the first character's category code is 0 (escape) and the second character's category code is not 11 (letter) is taken for the identification of a control symbol token and therefore the corresponding control symbol token will be "placed" into the "stream of tokens".

A sequence of characters in the input where the first character's category code is 0 (escape) and all trailing characters of that sequence have category code 11 (letter) is taken for the identification of a control word token and therefore the corresponding control word token will be "placed" into the "stream of tokens".

IIRC the "nameless" control sequence token, denoteable via \csname\endcsname or via a character whose category code is 0 (escape) at the end of a line of input in situations where no endline-character gets attached, is also a control word token.

Control sequence tokens that come into being as a result of expanding \csname..\endcsname or \ifcsname..\endcsname and whose names form character-sequences other than just a single non-catcode-11-character are also control word tokens.

After reading/tokenizing a control word token, TeX' reading apparatus will switch to state S (skipping blanks). When unexpanded-writing a control word token, TeX will write/attach an additional space character.

After reading/tokenizing a control symbol token, TeX' reading apparatus will switch to state M (middle of line). When unexpanded-writing a ontrol symbol token, TeX will not write/attach an additional space character.

(Be aware that treatment at writing-time of a one letter control sequence token either as a control word token or as a control symbol token depends on the category code of the letter/character in question at writing-time.)

The last thing that will be gathered with a \count-assignment is a <number>.

The components of quantities subsumed under the term <number> are explained in Donald E. Knuth's The TeXbook, Chapter 24: Summary of Vertical Mode.

When gathering/reading the components of a <number>, expansion takes place unless TeX is in a situation where TeX does not expand tokens. These situations are listed in Chapter 20: Definitions of The TeXbook.

Rakishly speaking you can say that the only thing that might be "consumed" in some cases of gathering components of a <number> is <one optional space>`.