1

In exercise 8.6 the characters ^^A, ^^B, ^^C, ^^M are defined to belong to categories 0, 7, 10, and 11 respectively. The question ist what tokens are produced from the input line ‘^^B^^BM^^A^^B^^C^^M^^@\M ’ (There is a space character after M). As a hint Knuth states that the line is followed by <return>, which is ^^M; and that ^^@ denotes the null character, which has category 9.

The solution is that there are 7 tokens: token list

Why isn't the space character after the control symbol ^^B ignored? Why does it not matter for the last control sequence that ^^M has category code 11?

Can anyone explain that to me, please?

1 Answer 1

2

The exercise assumes that \endlinechar=13, which corresponds to ^^M.

When a line of input is examined (but not yet tokenized), TeX will discard any trailing spaces along with the (system dependent) end-of-record character and insert the \endlinechar.

Hence, when tokenization begins, there is no trailing space and you get

^^B^^BM^^A^^B^^C^^M^^@\M^^M

The ^^ convention applies during tokenization, so you get

^^B7 ^^B7 ^^A0 ^^B7 ^^C10 ^^M11 ^^@9 \0 M11 ^^M11

However, note that tokens are formed one at a time, so TeX will not recognize two identical character of category code 7, because ^^B7 is followed by ^, so the ^^ convention now no longer applies and you get the first two tokens of category code 7, next a control symbol |^^B| (the bars stand for the box used in the TeXbook), a space token that's normalized as <space>10. Next ^^@9 is ignored and next a control word is formed |M^^M|.

2
  • 1
    So why is the space after the control symbol ^^B not skipped? On page 46 it reads “TEX goes into state S in case (c), or in case (b) with respect to a character of category 10.” Does case (b) not apply here?
    – horro
    Commented Feb 19, 2022 at 16:32
  • @horro Spaces after control symbols aren't ignored except in case of \ , when skipping spaces takes place. So with \^^B the “otherwise” case applies.
    – egreg
    Commented Feb 19, 2022 at 17:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .