'a^^Mb' isn't equivalent to 'a<return>b'; instead throws pdftex into an extended mode

Question

I'd expect (see background material below) the following Plain TeX manuscript

a^^Mb\bye

to be equivalent to

a
b\bye

and therefore to typeset thus:

a b

Instead, pdftex pauses mid-compilation, and writes the following to the console:

entering extended mode
(./test29.tex)
*

Why?

---

Background material

The TeXbook (20th printing, Addison-Wesley 1991) describes the meaning of ^^ and how it is processed as follows.

On p. 45:

TEX has a standard way to refer to the invisible characters of ASCII: Code 0 can be typed as the sequence of three characters ^^@, code 1 can be typed ^^A, and so on up to code 31, which is ^^_ (see Appendix C). If the character following ^^ has an internal code between 64 and 127, TEX subtracts 64 from the code; if the code is between 0 and 63, TEX adds 64. Hence code 127 can be typed ^^? [...] ^^ symbols can even be used as letters within control words.

[...]

There’s also a special convention in which ^^ is followed by two “lowercase hexadecimal digits,” 0–9 or a–f. With this convention, all 256 characters are are obtainable in a uniform way, from ^^00 to ^^ff. Character 127 is ^^7f.

Most of the ^^ codes are unimportant except in unusual applications. But ^^M is particularly noteworthy because it is code 13, the ASCII <return> that TEX normally places at the right end of every line of your input file.

On pp. 46-47:

If TEX sees a superscript character (category 7) in any state, and if that character is followed by another identical character, and if those two equal characters are followed by a character of code c < 128, then they are deleted and 64 is added to or subtracted from the code c. (Thus, ^^A is replaced by a single character whose code is 1, etc., as explained earlier.) However, if the two superscript characters are immediately followed by two of the lowercase hexadecimal digits 0123456789abcdef, the four-character sequence is replaced by a single character having the specified hexadecimal code. [...] After the replacement is made, TEX begins again as if the new character had been present all the time.

Answer 1

You are forgetting that TeX works on lines. By inserting ^^M part-way through your input line, all of the rest of it is ignored: TeX (at the macro level) never sees the B\bye so stops at the 'waiting for more input' prompt. Add \tracingall and perhaps a second \bye on a newline to see this.

Answer 2

Page 47 of the TeXbook, second doubly dangerous paragraph:

If TeX sees an end-of-line character (category 5), it throws away any other information that might remain on the current line. Then if TeX is in state N (new line), the end-of-line character is converted to the control sequence token ‘par’ (end of paragraph); if TeX is in state M (mid-line), the end-of-line character is converted to a token for character 32 (‘⍽’) of category 10 (space); and if TeX is in state S (skipping blanks), the end-of-line character is simply dropped.

The category code 5 character is normally inserted in the reading phase, when TeX absorbs a record (a line of input), but in the end any category code 5 character obeys the rule above.

The ^^ convention allows you to explicitly input a category code 5 character, which makes TeX apply the stated rule. Also ^^0d would do the same.

_{With par I denote what the TeXbook boxes, that is, the control sequence token with that name, as distinct from the primitive \par.}

---

TeX is not “thrown into an extended mode”: the note you see in the log file precedes any input, as you can infer from the open parenthesis that follows the note. It’s simply telling you that e-TeX extensions are being included (actually they have been included during format creation).

The asterisk denotes “interactive mode” and is the prompt for more input, because TeX has ended reading the file you specified on the command line, as can be inferred from the closed parenthesis after the file name. Had you called pdftex -interaction=nonstopmode test29.tex the console would have shown

entering extended mode
(./test29.tex)
! Emergency stop.
<*> test29

where <*> is telling the same information: TeX needed input and, being in non-stop mode, it halts.

Answer 3

---

Before explaining what is going on in your two examples I try to describe some aspects of how TeX processes a .tex-file:

There is an analogy in the TeXbook by Donald Knuth between TeX and an organism with eyes and a digestive tract:

1. Eyes ⇆ 2. Mouth ⇆ 3. Gullet ⇆ 4. Stomach and subsequent digestive organs

---

1. The eyes "look" at an entire line of a .tex-file and pre-process it:

   1. Each character is converted into TeX's internal character-representation-scheme (which either is ASCII or is unicode).
   2. All space-characters at the right end of the line are removed.
   3. A character is appended at the right end of the line whose coding-point's number in TeX's internal character-representation-scheme equals the value of the integer-parameter \endlinechar. The value of \endlinechar usually is 13 which denotes the ⟨carriage‑return character⟩.

   The eyes pass the entire resulting sequence of characters to the mouth in one go. The eyes do this only when the mouth requests characters of them.

2. The mouth is connected to the eyes and to the gullet. When the gullet requests a token, the mouth produces a token and delivers it to the gullet. In the process of producing the token the mouth uses/"chews" some of the characters that are inside it.
   When the mouth is empty and needs characters, it again requests a sequence of characters from the eyes. Then the eyes look at the next line of the .tex -file and pre-process it, and deliver the entire resulting sequence of characters to the mouth in one go.

   One possibility of how the mouth can become empty is the mouth encountering an end-of-line-character, i.e., a character whose current category code is 5: Then the mouth discards/throws away those characters that both stem from the line that was looked at/pre-processed by TeX's eyes as the last one and were not used yet for creating tokens.

   This possibility is mentioned in the TeXbook:

   If TeX sees an end-of-line character (category 5), it throws away any other information that might remain on the current line. Then if TeX is in state N (new line), the end-of-line character is converted to the control sequence token ‘par’ (end of paragraph); if TeX is in state M (mid-line), the end-of-line character is converted to a token for character 32 (‘⍽’) of category 10 (space); and if TeX is in state S (skipping blanks), the end-of-line character is simply dropped.

   What is done by the eyes is not related to TeX's concept of "tokens". What the eyes do is about the characters of the .tex-input-file. The next digestion-station, the mouth, is the first station where TeX's concept of "tokens" begins to play a rôle. You need to distinguish "characters" of the .tex-input-file from "character-tokens". The further exist in the .tex-file whose lines are "looked at" and pre-processed by TeX's eyes . The further, i.e. characters, exist as long as the .tex-file exists. As the result of the pre-processing by TeX's eyes copies of the further that are converted to TeX's internal character-representation scheme exist in TeX's mouth while TeX is running. The latter, i.e. character-tokens, are data-structures which are created in TeX's mouth according to the rules given by TeX's reading-apparatus and values of parameters like catcodes and \endlinechar. They consist of properties like a character-code and a category-code, and they exist only while TeX is running. Besides different kinds of character-tokens the mouth also produces control-sequence-tokens, i.e., these "backslash-thingies", e.g., \section or \LaTeX.

   Also note the subtle difference between the phrases "catcode" and "category-code". When using the phrase "catcode" I tend to refer to the set of rules which TeX's mouth obeys while creating tokens. You can change the rules via \catcode-assignments, e.g., after \catcode`\A=11\relax TeX's mouth will, when encountering the character A produce a character-token of category-code 11(letter), i.e. A11. When using the phrase "category-code" I tend to refer to the respective property of a character-token. I.e., the category-code of the character-token A11 is 11(letter). This category-code is cast in stone. It is fixed and doesn't change any more. It is 11 because at the time when TeX's mouth tokenized the corresponding A-character, the catcode for A-characters was 11.

3. The gullet expands expandable tokens. The gullet then requests tokens from the mouth when the stomach requests some from the gullet while the gullet itself is run out of tokens. The tokens that have passed the gullet or have come into being there by expansion end up in the stomach that requested them.

4. The stomach and subsequent digestive organs then use the tokens as directives for carrying out assignments, changing mode (horizontal/vertical/math), building boxes, lines, paragraphs, pages, creating messages and output files, etc.

---

Armed with this knowledge let's look at the one-liner:

a^^Mb\bye

TeX's eyes look at that line and pre-process it according to the rules described above. Pre-processing yields the following characters (not tokens yet!!!):

a^^Mb\bye⟨carriage‑return character⟩

These characters are passed on to TeX's mouth where tokens are produced one by one as needed:

a has catcode 11(letter), thus TeX's mouth produces an explicit character-token a11 and sends it down TeX's "gullet" for further processing. Also TeX's reading-apparatus switches to state M(mid-line). (The gullet is the place where expandable tokens get expanded. The tokens resulting from the gullet's work are sent to the stomach and further digestive organs as needed and boxes and pages are built, assignments are carried out, output-files and messages are created, etc.)

When a11 is processed, then TeX needs more tokens.

The remaining characters (not tokens yet) in TeX's mouth, stemming from TeX's eyes' pre-processing of the one-liner, are:

^^Mb\bye⟨carriage‑return character⟩

TeX's mouth takes the sequence ^^M for ^^-notation of a ⟨carriage‑return character⟩. The ⟨carriage‑return character⟩ usually has catcode 5(end of line).

TeXbook says about characters of catcode 5:

If TeX sees an end-of-line character (category 5), it throws away any other information that might remain on the current line. Then if TeX is in state N (new line), the end-of-line character is converted to the control sequence token ‘par’ (end of paragraph); if TeX is in state M (mid-line), the end-of-line character is converted to a token for character 32 (‘⍽’) of category 10 (space); and if TeX is in state S (skipping blanks), the end-of-line character is simply dropped.

So the remaining characters (not tokens yet) in TeX's mouth, stemming from TeX's eyes' pre-processing of the one-liner, are thrown away, and as TeX's reading-apparatus is in state M(mid-line) an explicit space-token (character-code 32, category code 10(space)) is sent down TeX's gullet. As an explicit space-token is sent down TeX's gullet, the reading-apparatus is switched to state S(skipping blanks). As "the remainder of the current line" is thrown away, this switching to state S(skipping blanks) is not important: TeX will have to start processing another line of input - hereby the reading-apparatus will be switched to state N(new line).

So b\bye⟨carriage‑return character⟩ is thrown away by TeX's mouth instead of being processed/instead of being transformed into tokens that would get processed further.

Now there is the following situation:

• TeX's eyes don't find any more lines of input in the .tex-input-file while the job is not finished yet and while there are no further .tex-input-files open for reading.

• As there are no characters pending that might stem from pre-processing a line of another .tex-input-file (which could be case when a line contained an \input-directive or the like followed by other stuff), TeX's mouth is empty.

Therefore TeX's eyes look at the console: In other words: TeX switches to interactive-mode so that the user can provide more lines of input by typing on the keyboard, e.g., commands for finishing the job, or commands for inputting other .tex-files, or whatever. Hereby in the beginning the reading-apparatus is switched to state N(new line).

---

---

Let's look at the two lines

a
b\bye

The first line is pre-processed. You get the character-sequence (not tokens yet!!!)

a⟨carriage‑return character⟩

Now tokenizing these characters begins:

a has catcode 11(letter), thus TeX produces an explicit character-token a11 and sends it down TeX's "gullet" for further processing. Also TeX's reading-apparatus switches to state M(mid-line).

The remaining characters of the line are:

⟨carriage‑return character⟩

The ⟨carriage‑return character⟩ usually has catcode 5(end of line).

TeXbook says about characters of catcode 5:

If TeX sees an end-of-line character (category 5), it throws away any other information that might remain on the current line. Then if TeX is in state N (new line), the end-of-line character is converted to the control sequence token ‘par’ (end of paragraph); if TeX is in state M (mid-line), the end-of-line character is converted to a token for character 32 (‘⍽’) of category 10 (space); and if TeX is in state S (skipping blanks), the end-of-line character is simply dropped.

So the remaining characters of the line (in this case only the ⟨carriage‑return character⟩) are thrown away and as TeX's reading-apparatus is in state M(mid-line) an explicit space-token (character-code 32, category code 10(space)) is sent down TeX's gullet. Then the reading-apparatus is switched to state S(skipping blanks).

The job is not finished yet and the end of the file is not reached yet.

So the second line gets pre-processed. You get the character-sequence (not tokens yet!!!)

b\bye⟨carriage‑return character⟩

The reading-apparatus is switched to state N(new line).
b has catcode 11(letter), thus TeX produces an explicit character-token b11 and sends it down TeX's "gullet" for further processing. Also TeX's reading-apparatus switches to state M(mid-line).

The remaining character-sequence (not tokens yet!!!) is:

\bye⟨carriage‑return character⟩

Due to the backslash being of catcode 0(escape) and therefore denoting that a control-sequence-token is to be gathered, and the following b being assigned catcode 11(letter) TeX starts gathering characters until either reaching a character whose catcode is not 11(letter) or reaching the last character of the line. The characters gathered are taken for the name of a control-word-token which then is sent down TeX's gullet for further processing.

Thus the control-word-token \bye is sent down TeX's gullet for further processing while the character-sequence ⟨carriage‑return character⟩ remains in TeX's mouth.

After tokenizing a control-word-token the reading-apparatus is switched to state S(skipping blanks).
The processing of the token \bye yields ending the job. Thus the ⟨carriage‑return character⟩ remaining in TeX's mouth is not processed.