# Question about TeXBook Exercise 8.6

In exercise 8.6 the characters `^^A`, `^^B`, `^^C`, `^^M` are defined to belong to categories 0, 7, 10, and 11 respectively. The question ist what tokens are produced from the input line `‘^^B^^BM^^A^^B^^C^^M^^@\M ’` (There is a space character after M). As a hint Knuth states that the line is followed by `<return>`, which is `^^M`; and that ^^@ denotes the `null` character, which has category 9.

The solution is that there are 7 tokens:

Why isn't the space character after the control symbol `^^B` ignored? Why does it not matter for the last control sequence that `^^M` has category code 11?

Can anyone explain that to me, please?

The exercise assumes that `\endlinechar=13`, which corresponds to `^^M`.

When a line of input is examined (but not yet tokenized), TeX will discard any trailing spaces along with the (system dependent) end-of-record character and insert the `\endlinechar`.

Hence, when tokenization begins, there is no trailing space and you get

``````^^B^^BM^^A^^B^^C^^M^^@\M^^M
``````

The `^^` convention applies during tokenization, so you get

`^^B`7 `^^B`7 `^^A`0 `^^B`7 `^^C`10 `^^M`11 `^^@`9 `\`0 `M`11 `^^M`11

However, note that tokens are formed one at a time, so TeX will not recognize two identical character of category code 7, because `^^B`7 is followed by `^`, so the `^^` convention now no longer applies and you get the first two tokens of category code 7, next a control symbol `|^^B|` (the bars stand for the box used in the TeXbook), a space token that's normalized as `<space>`10. Next `^^@`9 is ignored and next a control word is formed `|M^^M|`.

• So why is the space after the control symbol ^^B not skipped? On page 46 it reads “TEX goes into state S in case (c), or in case (b) with respect to a character of category 10.” Does case (b) not apply here? Feb 19 at 16:32
• @horro Spaces after control symbols aren't ignored except in case of `\ `, when skipping spaces takes place. So with `\^^B` the “otherwise” case applies. Feb 19 at 17:58