Before explaining what is going on in your two examples I try to describe some aspects of how TeX processes a .tex-file:
There is an analogy in the TeXbook by Donald Knuth between TeX and an organism with eyes and a digestive tract:
- Eyes ⇆ 2. Mouth ⇆ 3. Gullet ⇆ 4. Stomach and subsequent digestive organs
The eyes "look" at an entire line of a .tex-file and pre-process it:
- Each character is converted into TeX's internal character-representation-scheme (which either is ASCII or is unicode).
- All space-characters at the right end of the line are removed.
- A character is appended at the right end of the line whose coding-point's number in TeX's internal character-representation-scheme equals the value of the integer-parameter
\endlinechar
. The value of \endlinechar
usually is 13 which denotes the ⟨carriage‑return character⟩.
The eyes pass the entire resulting sequence of characters to the mouth in one go. The eyes do this only when the mouth requests characters of them.
The mouth is connected to the eyes and to the gullet. When the gullet requests a token, the mouth produces a token and delivers it to the gullet. In the process of producing the token the mouth uses/"chews" some of the characters that are inside it.
When the mouth is empty and needs characters, it again requests a sequence of characters from the eyes. Then the eyes look at the next line of the .tex -file and pre-process it, and deliver the entire resulting sequence of characters to the mouth in one go.
One possibility of how the mouth can become empty is the mouth encountering an end-of-line-character, i.e., a character whose current category code is 5: Then the mouth discards/throws away those characters that both stem from the line that was looked at/pre-processed by TeX's eyes as the last one and were not used yet for creating tokens.
This possibility is mentioned in the TeXbook:
If TeX sees an end-of-line character (category 5), it throws away any other information that might remain on the current line. Then if TeX is in state N (new line), the end-of-line character is converted to the control sequence token ‘par’ (end of paragraph); if TeX is in state M (mid-line), the end-of-line character is converted to a token for character 32 (‘⍽’) of category 10 (space); and if TeX is in state S (skipping blanks), the end-of-line character is simply dropped.
What is done by the eyes is not related to TeX's concept of "tokens". What the eyes do is about the characters of the .tex-input-file. The next digestion-station, the mouth, is the first station where TeX's concept of "tokens" begins to play a rôle. You need to distinguish "characters" of the .tex-input-file from "character-tokens". The further exist in the .tex-file whose lines are "looked at" and pre-processed by TeX's eyes . The further, i.e. characters, exist as long as the .tex-file exists. As the result of the pre-processing by TeX's eyes copies of the further that are converted to TeX's internal character-representation scheme exist in TeX's mouth while TeX is running. The latter, i.e. character-tokens, are data-structures which are created in TeX's mouth according to the rules given by TeX's reading-apparatus and values of parameters like catcodes and \endlinechar
. They consist of properties like a character-code and a category-code, and they exist only while TeX is running. Besides different kinds of character-tokens the mouth also produces control-sequence-tokens, i.e., these "backslash-thingies", e.g., \section
or \LaTeX
.
Also note the subtle difference between the phrases "catcode" and "category-code". When using the phrase "catcode" I tend to refer to the set of rules which TeX's mouth obeys while creating tokens. You can change the rules via \catcode
-assignments, e.g., after \catcode`\A=11\relax
TeX's mouth will, when encountering the character A
produce a character-token of category-code 11(letter), i.e. A11
. When using the phrase "category-code" I tend to refer to the respective property of a character-token. I.e., the category-code of the character-token A11
is 11(letter). This category-code is cast in stone. It is fixed and doesn't change any more. It is 11 because at the time when TeX's mouth tokenized the corresponding A
-character, the catcode for A
-characters was 11.
The gullet expands expandable tokens. The gullet then requests tokens from the mouth when the stomach requests some from the gullet while the gullet itself is run out of tokens. The tokens that have passed the gullet or have come into being there by expansion end up in the stomach that requested them.
The stomach and subsequent digestive organs then use the tokens as directives for carrying out assignments, changing mode (horizontal/vertical/math), building boxes, lines, paragraphs, pages, creating messages and output files, etc.
Armed with this knowledge let's look at the one-liner:
a^^Mb\bye
TeX's eyes look at that line and pre-process it according to the rules described above. Pre-processing yields the following characters (not tokens yet!!!):
a^^Mb\bye⟨carriage‑return character⟩
These characters are passed on to TeX's mouth where tokens are produced one by one as needed:
a
has catcode 11(letter), thus TeX's mouth produces an explicit character-token a11
and sends it down TeX's "gullet" for further processing. Also TeX's reading-apparatus switches to state M(mid-line). (The gullet is the place where expandable tokens get expanded. The tokens resulting from the gullet's work are sent to the stomach and further digestive organs as needed and boxes and pages are built, assignments are carried out, output-files and messages are created, etc.)
When a11
is processed, then TeX needs more tokens.
The remaining characters (not tokens yet) in TeX's mouth, stemming from TeX's eyes' pre-processing of the one-liner, are:
^^Mb\bye⟨carriage‑return character⟩
TeX's mouth takes the sequence ^^M
for ^^
-notation of a ⟨carriage‑return character⟩. The ⟨carriage‑return character⟩ usually has catcode 5(end of line).
TeXbook says about characters of catcode 5:
If TeX sees an end-of-line character (category 5), it throws away any other information that might remain on the current line. Then if
TeX is in state N (new line), the end-of-line character is converted
to the control sequence token ‘par’ (end of paragraph); if TeX is in
state M (mid-line), the end-of-line character is converted to a token
for character 32 (‘⍽’) of category 10 (space); and if TeX is in state
S (skipping blanks), the end-of-line character is simply dropped.
So the remaining characters (not tokens yet) in TeX's mouth, stemming from TeX's eyes' pre-processing of the one-liner, are thrown away, and as TeX's reading-apparatus is in state M(mid-line) an explicit space-token (character-code 32, category code 10(space)) is sent down TeX's gullet. As an explicit space-token is sent down TeX's gullet, the reading-apparatus is switched to state S(skipping blanks). As "the remainder of the current line" is thrown away, this switching to state S(skipping blanks) is not important: TeX will have to start processing another line of input - hereby the reading-apparatus will be switched to state N(new line).
So b\bye⟨carriage‑return character⟩
is thrown away by TeX's mouth instead of being processed/instead of being transformed into tokens that would get processed further.
Now there is the following situation:
TeX's eyes don't find any more lines of input in the .tex-input-file while the job is not finished yet and while there are no further .tex-input-files open for reading.
As there are no characters pending that might stem from pre-processing a line of another .tex-input-file (which could be case when a line contained an \input
-directive or the like followed by other stuff), TeX's mouth is empty.
Therefore TeX's eyes look at the console: In other words: TeX switches to interactive-mode so that the user can provide more lines of input by typing on the keyboard, e.g., commands for finishing the job, or commands for inputting other .tex-files, or whatever. Hereby in the beginning the reading-apparatus is switched to state N(new line).
Let's look at the two lines
a
b\bye
The first line is pre-processed. You get the character-sequence (not tokens yet!!!)
a⟨carriage‑return character⟩
Now tokenizing these characters begins:
a
has catcode 11(letter), thus TeX produces an explicit character-token a11
and sends it down TeX's "gullet" for further processing. Also TeX's reading-apparatus switches to state M(mid-line).
The remaining characters of the line are:
⟨carriage‑return character⟩
The ⟨carriage‑return character⟩ usually has catcode 5(end of line).
TeXbook says about characters of catcode 5:
If TeX sees an end-of-line character (category 5), it throws away any other information that might remain on the current line. Then if
TeX is in state N (new line), the end-of-line character is converted
to the control sequence token ‘par’ (end of paragraph); if TeX is in
state M (mid-line), the end-of-line character is converted to a token
for character 32 (‘⍽’) of category 10 (space); and if TeX is in state
S (skipping blanks), the end-of-line character is simply dropped.
So the remaining characters of the line (in this case only the ⟨carriage‑return character⟩) are thrown away and as TeX's reading-apparatus is in state M(mid-line) an explicit space-token (character-code 32, category code 10(space)) is sent down TeX's gullet. Then the reading-apparatus is switched to state S(skipping blanks).
The job is not finished yet and the end of the file is not reached yet.
So the second line gets pre-processed. You get the character-sequence (not tokens yet!!!)
b\bye⟨carriage‑return character⟩
The reading-apparatus is switched to state N(new line).
b
has catcode 11(letter), thus TeX produces an explicit character-token b11
and sends it down TeX's "gullet" for further processing. Also TeX's reading-apparatus switches to state M(mid-line).
The remaining character-sequence (not tokens yet!!!) is:
\bye⟨carriage‑return character⟩
Due to the backslash being of catcode 0(escape) and therefore denoting that a control-sequence-token is to be gathered, and the following b
being assigned catcode 11(letter) TeX starts gathering characters until either reaching a character whose catcode is not 11(letter) or reaching the last character of the line. The characters gathered are taken for the name of a control-word-token which then is sent down TeX's gullet for further processing.
Thus the control-word-token \bye
is sent down TeX's gullet for further processing while the character-sequence ⟨carriage‑return character⟩
remains in TeX's mouth.
After tokenizing a control-word-token the reading-apparatus is switched to state S(skipping blanks).
The processing of the token \bye
yields ending the job. Thus the ⟨carriage‑return character⟩
remaining in TeX's mouth is not processed.