# Conversion of space characters into space tokens

I'm trying to understand how the input, the expansion, and the execution processors work together. Running TeX on

\def\foo#1{(#1)\baz}%
\def\baz{baz}%
\foo{bla} Bar
\bye


yields "(bla)baz Bar" with a space between baz and Bar. Naively I would think that \foo is expanded, and after the expansion I should have

\foo{bla} Bar(bla)\baz Bar

with \baz now gobbling the space after it. Maybe, in the definition of \foo, \baz has been already tokenized and spaces are not gobbled after it, but that would mean that the execution processor has already processed the first line. Or maybe the whole line has been read first, in which case the space character between } and B has been already converted into a space token. Or maybe something else which I don't see :-)

For which reason isn't the space gobbled?

• Speaking in terms of TeX's anatomy, the space after a csname is seen and ignored by TeX's eyes (input reading), and only after that TeX's mouth (macro expansion) expands \foo and \bar. But at that time the space after the } is already a space token in the token stream, and the eyes will not "ignore" once more. I'm sure there is a more accurate way to put that – Phelype Oleinik Nov 12 at 10:51
• Your first suggestion is correct "has been already tokenized and spaces are not gobbled after" but I don't understand your doubt about the first line "but that would mean that the execution processor has already processed the first line" er yes, that line is first and tex reads line by line, I didn't understand that comment (or address it in my answer below) – David Carlisle Nov 12 at 11:56
• @campa no the line is read in one sense into an input buffer (and line ends normalised, to get cross platform behaviour) but that is without tokenization and is essentially an invisible implementation detail, tokenization then starts at the beginning of the line interleaved with execution, typesetting or anything else that happens, you can see this by making a text error such as an undefined command, the line break in teh error message shows how far along the line tex has read – David Carlisle Nov 12 at 12:22
• if the whole line was tokenized first you would not be able to do \catcode\!=13 \def!{whatever} as the tokenization of ! depends on executing that catcode assignment earlier in the line – David Carlisle Nov 12 at 12:24
• @DavidCarlisle Of course, obvious... ok, I'll have to think about it a little. – campa Nov 12 at 12:26

Usually TeX does process input line by line:

The whole line is read and the whole line is pre-processed.

• One step of pre-processing the whole line is:
All characters of the character-sequence which forms that line are converted from the computer-platform's character-representation-scheme to the TeX-engine's internal character-representation-scheme.
The computer-platform's character-representation-scheme might be whatsoever character-encoding. With modern computers this often is Unicode (and often the transformation-format is UTF-8). With older machines, e.g., running under MS-DOS, this might be some 8bit encoding/byte-encoding whereof ASCII (American Standard Code for Information Interchange) is a subset; e.g., when running under Win95/Win98/NT, this might be, e.g., Windows-1252 or iso-8859-1/iso-8859-15 or whatever.
With traditional TeX-engines the TeX-engine's internal character-representation-scheme is ASCII. With XeTeX- and with LuaTeX-enginges the TeX-engine's internal character-representation-scheme is Unicode (whereof ASCII is a subset).
• Another step of pre-processing the whole line is:
All spaces, i.e., all characters whose code-point has the number 32 in the TeX-engine's internal character-representation-scheme/in ASCII/in Unicode, that occur at the right end of the line, get removed.
• Yet another step of pre-processing the whole line is:
At the right end of the line a character gets appended whose code-point-number in the TeX engine's internal character-representation-schmeme equals the value of the integer-parameter \endlinechar.
• The reading-apparatus is switched to state N(new line).

After pre-processing TeX starts tokenizing the pre-processed line.

This means TeX "looks" at the pre-processed line character by character and hereby takes the character-sequence as a set of directives for appending tokens to the token-stream. Hereby category-codes of characters play a rôle.

["Looking" at the pre-processed line character by character and appending tokens to the token-stream takes place "on demand", i.e., only when TeX needs tokens while the token-stream is empty. E.g., when the token-stream is empty while gathering macro-arguments or a ⟨balenaced text⟩, or when "looking" whether there is more work to do as no command for ending the job—something like (plain TeX) \bye or \end or (LaTeX) \stop or \end{document}—has been encountered yet.
On the one hand assigning another value to the integer-parameter \endlinechar does affect the pre-processing of lines of input. Thus an assignment to \endlinechar does not affect the line of input wherein it occurs (but only subsequent lines) because obviously that line is already pre-processed at the time when the assignment is carried out.
On the other hand changing category-codes may affect tokenization of things while tokenization takes place on demand after pre-processing. Therefore changing category-codes might affect the tokenizing of things that (even in the current line) appear right after the assignment for changing the category-codes.
Changing the category-code of the "endline-character" may affect how the (during the pre-processing of the current line already appended) "endline-character" of the current line gets tokenized.

You can, e.g., type "I must not talk in class!" ten times by assigning \endlinechar a nice value and making the corresponding character active and defining that active character to deliver a horizontal box holding the phrase "I must not talk in class!" and then adding ten empty lines to the .tex-input (by hitting return ten times while typeing the sourecode), yielding the insertion of ten endline-characters during compilation as each of these ten empty lines gets pre-processed—notice that the \endlinechar-assignment does not affect the line wherein it occurs (but only subsequent lines) because that line is already pre-processed at the time when that \endlinechar-assignment is carried out. Each of the ten inserted endline-characters in turn gets tokenized as the mentioned active character delivering the horizontal box with the phrase "I must not talk in class!" :

\begingroup
%  Let's make 'A' active:
\catcode\A=13 %
% Let's have a scratch-counter for counting how many times
% the phrase "I must not talk in class!" is written:
\newcount\scratchcount
% Let's define the active-'A' to do some counting and to
% deliver the line "I must not talk in class!":
\def A{%
% Ensure vertical mode:
\ifvmode\else\par\fi
% Increment the scratch-counter and place the line/
% the horizontal box:
\advance\scratchcount by 1 %
\hbox{\number\scratchcount.\null\ I must not talk in class!}%
}%
% Make the character 'A' the endline-character:
\endlinechar=\A\relax
% (The \endlinechar-assignment in the line above does not affect
% that line. It does affect subsequent lines only. It does not
% lead to appending the character 'A' to that line as at the time
% of carrying out that assignment in TeX's stomach, that line is
% already pre-processed with the old value of \endlinechar (which
% is 13, denoting the return-character) ).
%
% Now let's have ten empty lines, yielding ten endline-characters
% 'A' whereof each gets tokenized as active-'A' expanding to the
% directives for doing some counting and delivering the line with
% the phrase "I must not talk in class!".

\endgroup%
% The comment-char at the end of the line above must be as the line
% above obviously gets pre-processed _before_ carrying out \endgroup
% and thus it also will have an endlinechar-'A' appended.
% Without the comment-char that 'A' would--as at the time of gathering
% the characters that form the name of the control-word-token '\endgr...'
% the  character 'A' is not of category-code 11(letter)--not be taken for
% something that belongs to the name of that "\endgr..."-control-word-token
% and therefore would trigger termination of gathering the name of the
% '\endgr...'-control-word-token and would be put back into the input
% stream.
% After processing/carrying out the control-word-token '\endgroup', 'A'
% is of category-code 11(letter).
% Therefore processing/tokenizing the 'A' that was put back into the
% input-steam would yield an 'A'-character-token of category-code
% 11(letter), at some later stage of processing yielding a glyph 'A'
% within the output-file/within the .dvi- or .pdf-file.
%
% Now let's get the token '\bye' in a funny way:
\endlinechar=e
\by


]

Let's look at your code:

Line 1:  \def\foo#1{(#1)\baz}%
Line 2:  \def\baz{baz}%
Line 3:  \foo{bla} Bar
Line 4:  \bye


Line 1 and 2 are code-lines without spaces, thus here no space-tokens come into being. We don't go into details here. Each of these lines ends with a percent-character while the percent-character has category-code 14(comment). With each of these lines due to the integer-parameter \endlinechar having the value 13 (13 denotes the return character in the TeX-engine's internal character-representation-scheme/in ASCII/in Unicode) a return-character will be appended behind that percent-character at the stage of pre-processing. But at the stage of tokenizing, characters of category-code 14(comment) (when not being taken for the name of a control-symbol-token) cause TeX to cease tokenizing the current line of input and to start processing the next line of input if present. Thus a percent-character within a line of input doesn't yield appending a token to the token-stream at all but causes TeX to silently "drop" it and that line of input's remainig characters. As the return-character appended due to \endlinechar also belongs to the remaining characters of that line of input, it is silently dropped as well.

Line 3 is pre-processed (by TeX's eyes) as follows:

The line is read and its single characters are converted to the TeX-engine's internal character-representation-scheme.

There are no spaces at the right end of the line. Thus there are no spaces at the right end of the line to remove.

Due to \endlinechar (usually) having the value 13 while 13 is the number of the code-point of the return-character in ASCII/in Unicode/in the TeX-engine's internal character-representation-scheme, (usually) a return-character is inserted behind the last character of the line, which is r. Usually the return-character has the category-code 5(end of line).

When TeX (in its mouth) begins tokenizing the pre-processed line, the reading-apparatus is switched to state N(new line).
(When the reading-apparatus is in state N(new line), then

• space-characters do not yield appending tokens to the token-stream at all but are simply dropped, and
• a character of category-code 5(end of line) yields appending the control-word-token \par to the token-stream and also causes TeX to cease tokenizing the remaining characters of the current line/and also causes TeX to drop the remaining characters of the current line and to start processing the next line of input if present.

)

Thus TeX's mouth by and by, i.e., whenever tokens are needed, tokenizes the pre-processed line/the pre-processed input-character-sequence (now converted to the TeX-engine's internal character-representation-scheme)

\foo{bla}⟨space-character⟩Bar⟨return-character⟩

as follows:

• Control-word-token \foo. (After appending a control-word-token to the token-stream, the reading-apparatus is switched to state S(skipping blanks).)

As \foo is a macro which processes an argument, the argument needs to be obtained by tokenizing some more input:

• Explicit character-token { (opening curly brace) of category-code 1(begin group). (After appending an explicit character-token which is not of category-code 10(space) or after appending a control-symbol-token differing from control-space (\ ), the reading-apparatus is switched to state M(middle of line).)

• Explicit character-token b of category-code 11(letter). (After appending an explicit character-token which is not of category-code 10(space) or after appending a control-symbol-token differing from control-space (\ ), the reading-apparatus is switched to state M(middle of line).)
• Explicit character-token l of category-code 11(letter). (After appending an explicit character-token which is not of category-code 10(space) or after appending a control-symbol-token differing from control-space (\ ), the reading-apparatus is switched to state M(middle of line).)
• Explicit character-token a of category-code 11(letter). (After appending an explicit character-token which is not of category-code 10(space) or after appending a control-symbol-token differing from control-space (\ ), the reading-apparatus is switched to state M(middle of line).)
• Explicit character-token } (closing curly brace) of category-code 2(end group). (After appending an explicit character-token which is not of category-code 10(space) or after appending a control-symbol-token differing from control-space (\ ), the reading-apparatus is switched to state M(middle of line).)

• Thus the following tokens are now send down from TeX's mouth to TeX's stomach—on their way to the stomach the tokens go through TeX's gullet, where expansion takes place:
\foo(control-word-token){1(begin-group)b11(letter)l11(letter)a11(letter)}2(end group)
while TeX's mouth still holds the remaining pre-processed input-character-sequence
⟨space-character⟩Bar⟨return-character⟩.

• Expansion of these tokens while passing TeX's gullet yields:

\foo requires a non-delimited argument. Explicit space-tokens preceding a non-delimited macro argument get discarded while gathering the tokens that form the argument. (A non-delimited argument either is a single token (which neither is an explicit space-token nor is an explicit character-token of category-code 1(begin group) nor is an explicit character-token of category-code 2(end group) nor is an \outer-token) or consists of a pair of matching curly braces (opening brace and closing brace) wherein a brace-balanced set of non-\outer-tokens is nested. That brace-balanced set of tokens can be "empty".) If present, a pair of matching curly braces that surrounds an entire macro argument (be it a delimited or a non-delimited macro argument) gets discarded when delivering the macro's replacement-text.
Expansion of \foo yields the following replacement:

(12(other)b11(letter)l11(letter)a11(letter))12(other)\baz(control-word-token)

The mouth still holds the remaining pre-processed input-character-sequence
⟨space-character⟩Bar⟨return-character⟩.

• While these tokens are slipping down the gullet, the expandable control-word-token \baz gets expanded as well—the following tokens reach the stomach of TeX:

(12(other)b11(letter)l11(letter)a11(letter))12(other)b11(letter)a11(letter)z11(letter)

Processing these tokens in the stomach (where assignments take place and boxes are bulit and paragraphs are split across lines and lines are placed on pages etc) yields switching to horizontal mode and to adding the glyph-sequence
(bla)baz
to the horizontal list from which the next line of text for the output-file/the .pdf-file is to be constructed.

TeX's mouth still holds the remaining pre-processed input-character-sequence
⟨space-character⟩Bar⟨return-character⟩.

• There is no indication that the job is to be finished, so TeX keeps its digestive processes going:

The reading-apparatus is neither in state N(new line) nor in state S(skipping blanks) but is in state M(middle of line) and TeX is not gathering the name of a control-symbol-token. So from the remaining pre-processed input-character-sequence in its mouth
⟨space-character⟩Bar⟨return-character⟩
it tokenizes the ⟨space-character⟩ as an explicit space-token (character-code 32, category-code 10(space)) and appends that to the token-stream/sends that down its gullet towards the stomach.
(After appending an explicit character-token of category-code 10(space) or after appending a control-space (\ ), the reading-apparatus is switched to state S(skipping blanks).)
As TeX is in horizontal mode, the space-token in the stomach causes TeX to add horizontal glue to the horizontal list which in turn (if not discarded for some reason) yields visible horizontal empty space in the .pdf-output-file.

TeX's mouth holds the remaining pre-processed input-character-sequence
Bar⟨return-character⟩.

• There is no indication that the job is to be finished, so TeX keeps its digestive processes going:

From the remaining pre-processed input-character-sequence in its mouth it tokenizes the explicit character-token B of category-code 11(letter) and sends that down its gullet toward the stomach. (After appending an explicit character-token which is not of category-code 10(space) or after appending a control-symbol-token differing from control-space (\ ), the reading-apparatus is switched to state M(middle of line).)

TeX's mouth holds the remaining pre-processed input-character-sequence
ar⟨return-character⟩.

• There is no indication that the job is to be finished, so TeX keeps its digestive processes going:

From the remaining pre-processed input-character-sequence in its mouth it tokenizes the explicit character-token a of category-code 11(letter) and sends that down its gullet toward the stomach. (After appending an explicit character-token which is not of category-code 10(space) or after appending a control-symbol-token differing from control-space (\ ), the reading-apparatus is switched to state M(middle of line).)

TeX's mouth holds the remaining pre-processed input-character-sequence
r⟨return-character⟩.

• There is no indication that the job is to be finished, so TeX keeps its digestive processes going:

From the remaining pre-processed input-character-sequence in its mouth it tokenizes the explicit character-token r of category-code 11(letter) and sends that down its gullet toward the stomach. (After appending an explicit character-token which is not of category-code 10(space) or after appending a control-symbol-token differing from control-space (\ ), the reading-apparatus is switched to state M(middle of line).)

TeX's mouth holds the remaining pre-processed input-character-sequence
⟨return-character⟩.

• There is no indication that the job is to be finished, so TeX keeps its digestive processes going:

As TeX is not gathering the name of a control-symbol-token and as the reading-apparatus is in state M(middle of line) while the return-character has category-code 5(end of line), TeX will append to the token-stream and send down its gullet an explicit space-token (character-code 32, category-code 10(space)).

(If TeX were encountering a character of category-code 5(end of line) while the reading-apparatus was in state N(new line) and TeX was not gathering the name of a control-symbol-token, then TeX would append the control-word-token \par to the token-stream.
That's why under normal circumstances

• empty lines in the source-code and
• lines in the source-code containing nothing but space-characters and
• lines in the source-code containing nothing but a mixture of characters of category-code 9(ignore) and 10(space), that mixture probably trailed by some space-characters

yield the control-word-token \par. (With each of these cases none of the characters (if present) in that line leads to inserting a token into the token-stream, thus the reading-apparatus still is in state N when encountering the return-character of category code 5(end of line) that got inserted due to the value of \endlinechar at the right end of the line at the stage of pre-processing the line.)

If TeX were encountering a character of category-code 5(end of line) while the reading-apparatus was in state S(skipping blanks) and TeX was not gathering the name of a control-symbol-token, then TeX would not append a token at all to the token-stream.)

When encountering a character of category-code 5(end of line) while not gathering the name of a control-symbol-token TeX in any case ceases tokenizing the current line, i.e., drops any remaining characters on the current line, and starts processing the next line if present.

• There is no indication that the job is to be finished, so TeX keeps its digestive processes going:
There are no more characters left in the mouth, so TeX's eyes start pre-processing the next line of input. The reading-apparatus is switched to state N(new line). The single characters of the pre-processed line go to TeX's mouth on demand where tokens are formed on demand. Tokens are sent from TeX's mouth towards TeX's stomach on demand. Hereby they pass TeX's gullet where expandable tokens get expanded/get replaced by their replacement-text. In the stomach assignments take place and boxes are bulit and paragraphs are split across lines and lines are placed on pages etc...

• OK, first thanks for the incredibly detailed answer (I can +1 only once though...). However, one thing is still uncear to me: you describe the process as if the whole line would be tokenized first, with expansion taking place afterwards. This seems to collide with David's comment above. – campa Nov 12 at 20:22
• @campa Unless I misread Ulrich's answer, the first phase he described is not tokenization, it is turning system-dependent character encoding into TeX's internal encoding [based on ASCII; what Knuth calls “internal character code”]. The output of this phase is a stream of characters in a system-independent encoding (“based on ASCII”), but not a stream of tokens. You can think of this phase as a kind of inputenc, but (more or less) hardwired into the TeX installation, not based on LaTeX and happening earlier in the TeX workflow. Seems related to the TCX files mentioned in texdoc web2c. – frougon Nov 12 at 22:27
• @campa I added a few remarks in order to point out the distinguishing between TeX's pre-processing of an entire line line of input and TeX's (on demand) tokenizing of the pre-processed line by "looking on demand" at the pre-processed-line character by character when needing tokens. – Ulrich Diez Nov 13 at 11:21
• @UlrichDiez The way you assembled the \bye token using \endlinechar illustrates very well that the \endlinechar-replacement produces an internal character, not a token, and that this happens before tokenization. Too bad I can only upvote once. :-) – frougon Nov 13 at 12:13
• @frougon Yes. The internal character-representation-scheme is "hardwired" in the TeX-engine. With traditional engines, this is ASCII. With XeTeX-/LuaTeX-based engines, this is Unicode. Assume a traditional TeX-engine running on an EBCDIC-platform where all text-files are encoded in EBCDIC-encoding. TeX with each line transforms EBCDIC-representation of characters to ASCII-representation before doing anything else. This transformation is the reason why the package inputenc does not work with TeX-installations on computer-platforms whose encoding does not have the ASCII-encoding as a subset. – Ulrich Diez Nov 13 at 20:56

characters are normally tokenized into a character token, using the current catcode settings, but after a character of catcode 0 is seen, it is not tokenized and the following characters are used to make a csname token.

In this case the following character is b of catcode 11 so tex will read all the following catcode 11 characters upto and including the first non-catcode 11 character, or end of line.

So here the sequence of catcode 11 characters is baz and will make a csname token with name baz the non-catcode11 character that was used to terminate the csname scan is returned to the input stream (as a character, still untokenised) unless it it is catcode 10 space character in which case it is discarded, and tex goes into its skipping blanks state, so that any following spaces are also discarded. If the scan was terminated by end of line then tex goes straight to its beginning of line state without adding the token that usually produces a space at ends of lines, and all spaces at the beginning of the next line will be discarded as usual.

so in your case the characters after \baz are } in the first definition, { in the second definition so no special space handling is involved, just in your later suggested use of explicit (bla)\baz Bar the non-catcode 11 character is a space and discarded.

When macros are expanded the replacement texts are list of tokens so none of this character to token or catcode lookup is involved at all.

Let me modify your code

\def\foo#1{(#1)\baz}
\def\baz{baz}

\foo{bla} Bar\baz Gnu

\bye


The definitions are actually irrelevant. When TeX reads the input it tokenizes it; so let's count the tokens in the relevant line:

\foo{1b11l11a11}2⍽10 • B11a11r11\bazG11n11u11⍽10

I also added the category codes, when possible; control sequence tokens have no category code. The last space token is generated by the end line.

There is no space token after \baz, because spaces are ignored after control words during the process of tokenization.

Now TeX starts expanding macros, starting from the left. Since \foo is a one-argument macro and is followed by {1, the argument is everything up to the matching }2. Thus TeX removes all these tokens and replaces them with the replacement text saved at definition time:

(12b11l11a11)12\baz⍽10 • B11a11r11\bazG11n11u11⍽10

The tokens up to \baz are passed to the next stage, leaving

\baz⍽10 • B11a11r11\bazG11n11u11⍽10

Now \baz is a no-argument macro, so no lookup for undelimited arguments is done, which would ignore spaces; the replacement leaves

b11a11z11⍽10 • B11a11r11\bazG11n11u11⍽10

Note that TeX is not doing tokenization at this stage, so spaces after control sequences are not ignored.

When the macro replacement is performed, TeX uses already formed tokens; so \baz at the beginning of the third shown token list is actually the “internal” representation of the token. A following space is not ignored.

This is necessary. Suppose you have

\def\foo#1{#1 is good}
\def\egreg{EG}


Then you want that \foo{EG} or \foo\egreg print the same, independently of what's the argument passed to \foo. The parameter in the definition is followed by a space, so also after macro replacement there will be a space.

Note, The description above is a simplification of what really happens. The line is not tokenized immediately: only the part of the line that's needed is scanned. So TeX actually starts tokenizing \foo and after having found a one-argument macro, it looks for what comes along, which is an open brace, so TeX tokenizes up to finding the matching closed brace. And so on. However, since there is no category code change involved, pretending that TeX tokenizes at once the whole line is not the truth, but a good approximation to it for the task at hand.

What would be the problem in immediately tokenizing a line? Consider

\catcode?=\active ?


If the line were tokenized immediately, the ? would be assigned category code 12 and not 13. Instead, tokenizing when the need arises solves the problem. The second ? is tokenized after the category code assignment has been performed.

• I was wondering why you hadn't answered yet ;-) Incidentally, what you write in the Note at the end is for me the most interesting bit. – campa Nov 13 at 16:54
• @campa It should have been fixed, now. – egreg Nov 13 at 17:12
• How many lexical categories does TeX have? – manooooh Nov 14 at 0:54
• @manooooh Not sure what you mean. – egreg Nov 14 at 7:25
• @manooooh TeX has a few keywords that, in some contexts are interpreted in a fixed way; for instance the keywords that can follow \hrule or \vrule are width, height and depth. But there's no real distinction like in C. – egreg Nov 14 at 12:14

In your source document, you wrote:

\foo{bla} Bar


TeX's eyes will turn that into tokens. Usually, you'll get 10 tokens. They are the control sequence whose name is foo and 9 character tokens. Every character token has a category. You'll get 6 letters, one begin-group, one end-group and a space.

If instead you wrote

\foo {bla}Bar


you'll only get 9 tokens. You won't get the space. That's because TeX's eyes ignore all spaces immediately after a control sequence, such as \foo. This turns out to be helpful.

I've explained how TeX works, but not why you're confused. TeX macros don't expand text to text. They expand tokens to tokens. It's not that TeX macros create a second source document, which is then re-read. I hope this helps.

Finally, there's a line ending character at the end of the line. That's another story. (One line ending is a space, two in a row are converted to a \par.)

This answer is based on Chapter 7, How TeX Reads What You Type, in Don Knuth's The TeXbook. It's also @egreg's answer, simplified to answer just your question.

Rereading this answer, I realise the phrase control sequence is used with two meanings. This might have contributed to your confusion. Let me clarify. Call \foo, followed by a non-letter, a control sequence literal. When read by TeX's eyes, it produces a control sequence token, whose name is foo.

In Python, for example, 'Hello world' is a string literal, which when compiled produces a string (whose value is 'Hello world'). There are of course other ways in Python to get a string. For example 'Hello' + 'world'. Similarly, TeX has the \csname ... \endcsname` commands.