6

Consider the following plain TeX manuscript

\catcode`z=10
\newcount\charcode
\charcode=`z
\font\myfont=cmtt14\myfont
character code = \the\charcode, category code = \the\catcode\charcode
\end

The manuscript starts by setting the category code of the letter z (ASCII 122) to 10. It then stores the letter z's character code in the counter \charcode. It then changes the font to Computer Modern Typewriter. Finally, it typesets the number stored in \charcode as well as the corresponding category code.

When this manuscript is compiled with pdftex, the resulting typeset text is

character code = 32, category code = 10

So it appears that changing the category code of the letter z also changed its character code from 122 to 32.

Is this behavior specified in the TeXbook or in the TeX engine's source code, or is it a bug in pdftex?

  • 4
    TeXbook, page 47, fourth doubly dangerous paragraph: the character code in a space token is always 32 – egreg Sep 26 '17 at 15:35
  • 1
    somewhat related: see 2.10.5 ⟨space token⟩ in TeX by Topic. TEX’s input processor converts (when in state M) all tokens with category code 10 into real spaces: they get character code 32. He gives the example of \catcode`Q=10 \def\q{aQb}\show\q. – user4686 Sep 26 '17 at 15:52
6

The TeXbook, page 47, fourth doubly dangerous paragraph:

If TeX sees a character of category 10 (space), the action depends on the current state. If TeX is in state N or S, the character is simply passed by, and TeX remains in the same state. Otherwise TeX is in state M; the character is converted to a token of category 10 whose character code is 32, and TeX enters state S. The character code in a space token is always 32.

This regards the tokenization phase. However, the book doesn't immediately tell the whole truth and this should be complemented with a discussion of “funny spaces” (pages 377–378):

The situation is surprisingly complex, because it's possible to use \uppercase to create “funny space” tokens like *10; for example, the commands

\uccode` =`* \uppercase{\uppercase{\def\fspace{ }\let\ftoken= } }

make \fspace a macro that expands to a funny space, and they make \ftoken an implicit funny space. (The tests \if\fspace*, \if\ftoken*, \ifcat\fspace\stoken|, and \ifcat\ftoken\stoken will all be true, assuming that * has category 12; but if * has category 10, \if\fspace* will be false, because TeX normalizes all newly created space tokens to 10, as explained in Chapter 8.) Since the various forms of space tokens are almost identical in behavior, there's no point in dwelling on the details.

Funny spaces can only be created with \uppercase or \lowercase, in various ways. So the statement “The character code in a space token is always 32“ is essentially true, but not the whole truth, because some esotericisms can enter the way.

  • Thanks. What is the purpose of the interior \uppercase? Won't the same effect be achieved with \uccode` =`* \uppercase{\def\fspace{ }\let\ftoken= }? – Evan Aad Sep 26 '17 at 17:31
  • 1
    @EvanAad Exercise for the reader. ;-) Hint: \let= accepts one optional space. – egreg Sep 26 '17 at 17:33
  • I see. How about \uccode`␣=`*\uppercase{\def\fspace{␣}\let\ftoken=␣␣}? Is this the same as the TeXbook example? – Evan Aad Sep 26 '17 at 17:38
  • 2
    @EvanAad ␣␣ becomes a single space token during tokenization. – egreg Sep 26 '17 at 17:40
  • 1
    @BrunoLeFloch “One optional space” is the terminology in the TeXbook. – egreg Sep 26 '17 at 22:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.