'a^^Mb' isn't equivalent to 'a<return>b'; instead throws pdftex into an extended mode

Question

I'd expect (see background material below) the following Plain TeX manuscript

a^^Mb\bye

to be equivalent to

a
b\bye

and therefore to typeset thus:

a b

Instead, pdftex pauses mid-compilation, and writes the following to the console:

entering extended mode
(./test29.tex)
*

Why?

---

Background material

The TeXbook (20th printing, Addison-Wesley 1991) describes the meaning of ^^ and how it is processed as follows.

On p. 45:

TEX has a standard way to refer to the invisible characters of ASCII: Code 0 can be typed as the sequence of three characters ^^@, code 1 can be typed ^^A, and so on up to code 31, which is ^^_ (see Appendix C). If the character following ^^ has an internal code between 64 and 127, TEX subtracts 64 from the code; if the code is between 0 and 63, TEX adds 64. Hence code 127 can be typed ^^? [...] ^^ symbols can even be used as letters within control words.

[...]

There’s also a special convention in which ^^ is followed by two “lowercase hexadecimal digits,” 0–9 or a–f. With this convention, all 256 characters are are obtainable in a uniform way, from ^^00 to ^^ff. Character 127 is ^^7f.

Most of the ^^ codes are unimportant except in unusual applications. But ^^M is particularly noteworthy because it is code 13, the ASCII <return> that TEX normally places at the right end of every line of your input file.

On pp. 46-47:

If TEX sees a superscript character (category 7) in any state, and if that character is followed by another identical character, and if those two equal characters are followed by a character of code c < 128, then they are deleted and 64 is added to or subtracted from the code c. (Thus, ^^A is replaced by a single character whose code is 1, etc., as explained earlier.) However, if the two superscript characters are immediately followed by two of the lowercase hexadecimal digits 0123456789abcdef, the four-character sequence is replaced by a single character having the specified hexadecimal code. [...] After the replacement is made, TEX begins again as if the new character had been present all the time.

Answer 1

You are forgetting that TeX works on lines. By inserting ^^M part-way through your input line, all of the rest of it is ignored: TeX (at the macro level) never sees the B\bye so stops at the 'waiting for more input' prompt. Add \tracingall and perhaps a second \bye on a newline to see this.

Answer 2

Page 47 of the TeXbook, second doubly dangerous paragraph:

If TeX sees an end-of-line character (category 5), it throws away any other information that might remain on the current line. Then if TeX is in state N (new line), the end-of-line character is converted to the control sequence token ‘par’ (end of paragraph); if TeX is in state M (mid-line), the end-of-line character is converted to a token for character 32 (‘⍽’) of category 10 (space); and if TeX is in state S (skipping blanks), the end-of-line character is simply dropped.

The category code 5 character is normally inserted in the reading phase, when TeX absorbs a record (a line of input), but in the end any category code 5 character obeys the rule above.

The ^^ convention allows you to explicitly input a category code 5 character, which makes TeX apply the stated rule. Also ^^0d would do the same.

_{With par I denote what the TeXbook boxes, that is, the control sequence token with that name, as distinct from the primitive \par.}

---

TeX is not “thrown into an extended mode”: the note you see in the log file precedes any input, as you can infer from the open parenthesis that follows the note. It’s simply telling you that e-TeX extensions are being included (actually they have been included during format creation).

The asterisk denotes “interactive mode” and is the prompt for more input, because TeX has ended reading the file you specified on the command line, as can be inferred from the closed parenthesis after the file name. Had you called pdftex -interaction=nonstopmode test29.tex the console would have shown

entering extended mode
(./test29.tex)
! Emergency stop.
<*> test29

where <*> is telling the same information: TeX needed input and, being in non-stop mode, it halts.