1

In exercise 8.6 the characters ^^A, ^^B, ^^C, ^^M are defined to belong to categories 0, 7, 10, and 11 respectively. The question ist what tokens are produced from the input line ‘^^B^^BM^^A^^B^^C^^M^^@\M ’ (There is a space character after M). As a hint Knuth states that the line is followed by <return>, which is ^^M; and that ^^@ denotes the null character, which has category 9.

The solution is that there are 7 tokens: token list

Why isn't the space character after the control symbol ^^B ignored? Why does it not matter for the last control sequence that ^^M has category code 11?

Can anyone explain that to me, please?

Improve this question

1 Answer 1

Reset to default
2

The exercise assumes that \endlinechar=13, which corresponds to ^^M.

When a line of input is examined (but not yet tokenized), TeX will discard any trailing spaces along with the (system dependent) end-of-record character and insert the \endlinechar.

Hence, when tokenization begins, there is no trailing space and you get

^^B^^BM^^A^^B^^C^^M^^@\M^^M

The ^^ convention applies during tokenization, so you get

^^B7 ^^B7 ^^A0 ^^B7 ^^C10 ^^M11 ^^@9 \0 M11 ^^M11

However, note that tokens are formed one at a time, so TeX will not recognize two identical character of category code 7, because ^^B7 is followed by ^, so the ^^ convention now no longer applies and you get the first two tokens of category code 7, next a control symbol |^^B| (the bars stand for the box used in the TeXbook), a space token that's normalized as <space>10. Next ^^@9 is ignored and next a control word is formed |M^^M|.

Improve this answer
2
  • So why is the space after the control symbol ^^B not skipped? On page 46 it reads “TEX goes into state S in case (c), or in case (b) with respect to a character of category 10.” Does case (b) not apply here?
    – horro
    Feb 19 at 16:32
  • @horro Spaces after control symbols aren't ignored except in case of \ , when skipping spaces takes place. So with \^^B the “otherwise” case applies.
    – egreg
    Feb 19 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.