3

On page 48 of The TeXbook, there is an exercise

Assume that the category codes of plain TEX are in force, except that the characters ^^A, ^^B, ^^C, ^^M belong respectively to categories 0, 7, 10, and 11. What tokens are produced from the (rather ridiculous) input line ^^B^^BM^^A^^B^^C^^M^^@\M␣ (the last character is a space)? (Remember that this line is followed by ⟨return⟩, which is ^^M; and recall that ^^@ denotes the ⟨null⟩ character, which has category 9 when INITEX begins.)

The answer is

The two ^^B’s are not recognized as consecutive superscript characters, since the first ^^B is converted to code 2 which doesn’t equal the following character ^. Hence the result is seven tokens: ^^B7 ^^B7 M11 |^^B| 10 ^^M11 |M^^M| . The last of these is a control word whose name has two letters. The ⟨space⟩ after \M is deleted before TeX inserts the ⟨return⟩ token.

Question 1. What is the code 2?

Question 2. How to undersand the category 7?

^^56\par
\catcode`\@=7
@@56\par
%@^10
\bye

produces

enter image description here

However, why does @^10 cause error if uncomment out the @^10?

Question 3. The space following the control symbol should be omitted. Why isn't the space following the ^^B omitted? ^^B^^BM^^A^^B^^C^^M^^@\M should be tokenized as ^^B7^^B7M11^^A0^^B7^^C10^^M11^^@9Mcontrol word 10^^M5.

As ^^A0^^B7 need to be converted to ^^Bcontrol symbol, so it is ^^B7^^B7M11^^Bcontrol symbol^^C10^^M11^^@9Mcontrol word 10^^M5.

As the space after the control symbol should be omitted, is ^^B7^^B7M11^^Bcontrol symbol^^M11^^@9Mcontrol word^^M5.

As category 9 should be omitted, it is ^^B7^^B7M11^^Bcontrol symbol^^M11Mcontrol word^^M5.

Question 4. Why is the answer ^^B7^^B7M11^^Bcontrol symbol 10^^M11M^^Mcontrol word?

5
  • B has ASCII code 66, so ^^B results in character code 2 (that's the "code 2").
    – Skillmon
    Nov 29, 2023 at 7:52
  • Your @^ doesn't result in a ^^ notation as the two tokens have different character codes. The two consecutive category 7 tokens starting that notation need to have the same character code as well. Hence the @ in @^10 is evaluated as a maths superscript, but used outside of maths this is an error (and inside of maths you'd have two consecutive superscripts which is an error as well).
    – Skillmon
    Nov 29, 2023 at 7:56
  • The space following ^^B isn't omitted, as that's no place where TeX ignores spaces. ^^B is just a token (the fact that TeX would ignore a space when searching for the token or braced group it should use as superscript doesn't mean the space isn't there during tokenisation). Also, spaces are only ignored after control words (control sequences consisting of one or more category code 11 tokens), not control symbols. Compare a\$ a with a\$a.
    – Skillmon
    Nov 29, 2023 at 7:58
  • You have chosen the most obscure exercise in TeXbook, congratulations. Note, I never needed to start to thinking about such a special cases and I am able to create good TeX macros without this. Note2: many exercises in TeXbook have similar obscurity. Maybe, it is better to leave them as historical curiosities.
    – wipet
    Nov 30, 2023 at 16:18
  • @wipet Maybe it's my personality. When I see something I don’t understand, I want to understand immediately.
    – Y. zeng
    Dec 1, 2023 at 2:15

2 Answers 2

6

Question 1

B is character code 66, ^^B is character code (66 - 64) = 2, that's the code 2.

Question 2

The two consecutive category code 7 tokens starting the ^^-notation need to have the same character code, ^^B^ might be two consecutive category 7 tokens, but the first is character code 2, the second 94, so they aren't interpreted as ^^ and changing the following token. The same is true for your @^10, those are two consecutive category 7 tokens, but with different character code, so they are interpreted as normal maths superscript, which errs outside of maths mode.

The ^^ needing to have the same character code can be found in this passage of the TeXbook (pay attention to "by another identical character"):

\ddanger If \TeX\ sees a superscript character (category 7) in any state,
and if that character is followed by another identical character, and if
those two equal characters are followed by a character of code
$c<128$, then they
are deleted and 64 is added~to or subtracted from the code~$c$.
\ (Thus, |^^A| is
replaced by a single character whose code is~1, etc., as explained earlier.) \
However, if the two superscript characters are immediately followed by two
of the lowercase hexadecimal digits |0123456789abcdef|, the
four-character sequence is replaced by a single character having the
specified hexadecimal code.
The replacement is carried out also if such a trio or quartet of
characters is encountered during steps (b) or~(c) of the control-sequence-name
scanning procedure described above. After the replacement is made, \TeX\
begins again as if the new character had been present all the time.
If a superscript character is not the first of such a trio or quartet, it is
handled by the following rule.

Question 3

Spaces are only ignored after control words (control sequences built of one or more category 11 tokens), not control symbols. \^^B is a control symbol (^^B has category 7), so spaces aren't ignored after it. Compare with the result of for instance a\$a\par a\$ a\par.

12
  • Thanks. 1. What is the meaning of ^^B is just a token (the fact that TeX would ignore a space when searching for the token or braced group it should use as superscript doesn't mean the space isn't there during tokenisation)? 2. How do you know this The two consecutive category code 7 tokens starting the ^^-notation need to have the same character code? 3. Why is the end of answer M^^M control sequence? I thought it should be M(control word)^^M(5).
    – Y. zeng
    Nov 29, 2023 at 8:19
  • @Y.zeng 1) in my comment I first overlooked the category 0 token in front of the ^^B, and that was for a single category code 7 token, but the same is true (more or less) for a control symbol (so category 0 token followed by category 7 token). 2) see edit. 3) The line ends in \M^^M (the ^^M is the new line character and of category 11), spaces at the end of a line are removed from the input stream early on.
    – Skillmon
    Nov 29, 2023 at 9:17
  • So, the last of the answer should be "|M|(control word)^^M(5)" I thought. Why does the author formulate it as |M^^M|(control word or symbol, I can't confirm this.)? Why do two tokens become one token?
    – Y. zeng
    Nov 29, 2023 at 9:49
  • @Y.zeng because ^^M has category 11, hence is a letter, hence the input during tokenisation shows <code>\\<sub>0</sub>M<sub>11</sub>^^M<sub>11</sub></code> which forms a single control word built from two letters (M and ^^M).
    – Skillmon
    Nov 29, 2023 at 9:52
  • But there is a problem. At the bottom of page 47 of The TeXbook, it states that an ignored character (should be category 9) can't appear in the midst of a control sequence name. From the post here, there a space (category 10) appear in the midst of a control sequence, as at the end of the characters there is a space and TeX adds ^^M at the end. Don't you think it's strange at this point? <null> cannot appear in the middle of the control sequence, but space can?
    – Y. zeng
    Nov 29, 2023 at 9:59
4

You want first to review how TeX transforms input (a sequence of bytes) into tokens. Let's consider the most common case, when TeX is receiving input from a file.

The operating system will feed the input one record at a time; what a record is depends on the OS, but in the most common cases it will be a line of text. At this point, the first operations are performed:

  1. the “end-of-record” signal (if any) is thrown away together with all trailing characters with code 32;
  2. at the end of the modified record, a character with code \endlinechar is appended.

Note that no assignment of category code has been done yet. The input processor (the eyes) is in state N (see page 46) and starts to examine the record character by character, comparing it with the \catcode table.

State N means that any character with category code 10 is ignored (this is how initial spaces or tabs get discarded). When some character not having category code 10 is found, the processor usually switches to state M (and this is where catcodes are assigned and control sequences are formed).

The next character is reexamined and assigned its catcode. Now the fun begins:

  1. if the catcode is 5 and the state was still N, TeX generates a |par| token and sends it to the expansion processor; otherwise TeX discards anything that can remain on the record and switches to state N;
  2. if the catcode is 9, the character is ignored;
  3. if the catcode is 1, 2, 3, 4, 6, 8, 11, 12, 13, the character is assigned its catcode and passed on as a character token to the expansion processor (the mouth);
  4. if the catcode is 0, a control sequence is formed;
  5. if the catcode is 7, more fun is involved.

In cases 2 to 5, the state becomes M.

Case 5 is what exercise 8.6 is dealing with and the ^^ convention enters the scene.

After seeing a character x with category code 7, TeX looks at the next character (no token has been formed yet); if the next character y hasn't the same character code and category code as x, a token with catcode 7 is passed to the expansion processor. Otherwise the business related to the ^^ convention is carried on (which I've explained in another answer to a question of yours).

Thus with

\catcode`_=7

^^B
__B
_^B

the third input would not trigger the convention, whereas the first two would. Note that this is essentially the same as your question 2:

^^56\par
\catcode`\@=7
@@56\par
@^10

The ^^56 in first line is converted to V as ^^56 in the third line; but in the fourth line the character codes are different, so they don't trigger the convention.

You may also input ^ as ^^5e using the convention. However

^^5e^^5eB

would not produce the same as ^^B, because the first combination ^^5e is effectively converted to a ^ character, but this would lead to ^^^5eB which is four characters. See the paragraph following the character table on page 368: you're not supposed to know the answer to exercise 8.6 until you have read also Appendix C.

Part of exercise 8.6 is similar to the above paragraph. You have

^^B^^BM^^A^^B^^C^^M^^@\M␣

where ^^A has catcode 0, ^^B has catcode 7, ^^C has catcode 10 and ^^M has (funnily) catcode 11.

The first ^^ triggers the convention (because ^ is assumed to have catcode 7), so the character with ASCII code 2 is formed and its category code is indeed 7. However it is followed by ^ which has catcode 7, but it's not the same! It's a similar situation to your @^10. Thus the expansion processor is passed the pair (2,7) (charcode and catcode) and TeX moves on; it finds ^ followed by ^, so the convention is triggered and another pair (2,7) is passed on.

Answering that ^^M is found would be wrong. The following M is passed on as the pair (77,11). Next ^^A is seen (with the convention working), which has catcode 0, so a control sequence is formed; the following character is ^^B (with the convention working) which has catcode 7, so a control symbol is obtained and the token |^^B| is passed on. Next ^^C is seen (with the convention); it has catcode 10, so it's normalized and the pair (32,10) is passed on; it's not ignored because it follows a control symbol. Then you find ^^M and the pair (13,11) is passed on. Next ^^@ is ignored and |M^^M| is formed.

There's a trick! The trailing space is not actually there, because it has been discarded when the record is examined, see the first rule at the beginning, and ^^M is added.

2
  • Thanks. How do you know this: the “end-of-record” signal (if any) is thrown away together with all trailing characters with code 32? 2. For this: otherwise TeX discards anything that can remain on the record and switches to state N, may you give an example? 3. How to get this 32 from it has catcode 10, so it's normalized and the pair (32,10) is passed on? 4. How to get this 13 hen you find ^^M and the pair (13,11) is passed on?
    – Y. zeng
    Nov 29, 2023 at 11:47
  • 1
    @Y.zeng Start rereading the TeXbook avoiding dangerous paragraphs. Everything is explained earlier in the book.
    – egreg
    Nov 29, 2023 at 13:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .