Consider the following plain TeX manuscript

character code = \the\charcode, category code = \the\catcode\charcode

The manuscript starts by setting the category code of the letter z (ASCII 122) to 10. It then stores the letter z's character code in the counter \charcode. It then changes the font to Computer Modern Typewriter. Finally, it typesets the number stored in \charcode as well as the corresponding category code.

When this manuscript is compiled with pdftex, the resulting typeset text is

character code = 32, category code = 10

So it appears that changing the category code of the letter z also changed its character code from 122 to 32.

Is this behavior specified in the TeXbook or in the TeX engine's source code, or is it a bug in pdftex?

  • 4
    TeXbook, page 47, fourth doubly dangerous paragraph: the character code in a space token is always 32
    – egreg
    Sep 26, 2017 at 15:35
  • 1
    somewhat related: see 2.10.5 ⟨space token⟩ in TeX by Topic. TEX’s input processor converts (when in state M) all tokens with category code 10 into real spaces: they get character code 32. He gives the example of \catcode`Q=10 \def\q{aQb}\show\q.
    – user4686
    Sep 26, 2017 at 15:52

1 Answer 1


The TeXbook, page 47, fourth doubly dangerous paragraph:

If TeX sees a character of category 10 (space), the action depends on the current state. If TeX is in state N or S, the character is simply passed by, and TeX remains in the same state. Otherwise TeX is in state M; the character is converted to a token of category 10 whose character code is 32, and TeX enters state S. The character code in a space token is always 32.

This regards the tokenization phase. However, the book doesn't immediately tell the whole truth and this should be complemented with a discussion of “funny spaces” (pages 377–378):

The situation is surprisingly complex, because it's possible to use \uppercase to create “funny space” tokens like *10; for example, the commands

\uccode` =`* \uppercase{\uppercase{\def\fspace{ }\let\ftoken= } }

make \fspace a macro that expands to a funny space, and they make \ftoken an implicit funny space. (The tests \if\fspace*, \if\ftoken*, \ifcat\fspace\stoken|, and \ifcat\ftoken\stoken will all be true, assuming that * has category 12; but if * has category 10, \if\fspace* will be false, because TeX normalizes all newly created space tokens to 10, as explained in Chapter 8.) Since the various forms of space tokens are almost identical in behavior, there's no point in dwelling on the details.

Funny spaces can only be created with \uppercase or \lowercase, in various ways. So the statement “The character code in a space token is always 32“ is essentially true, but not the whole truth, because some esotericisms can enter the way.

  • Thanks. What is the purpose of the interior \uppercase? Won't the same effect be achieved with \uccode` =`* \uppercase{\def\fspace{ }\let\ftoken= }?
    – Evan Aad
    Sep 26, 2017 at 17:31
  • 1
    @EvanAad Exercise for the reader. ;-) Hint: \let= accepts one optional space.
    – egreg
    Sep 26, 2017 at 17:33
  • I see. How about \uccode`␣=`*\uppercase{\def\fspace{␣}\let\ftoken=␣␣}? Is this the same as the TeXbook example?
    – Evan Aad
    Sep 26, 2017 at 17:38
  • 2
    @EvanAad ␣␣ becomes a single space token during tokenization.
    – egreg
    Sep 26, 2017 at 17:40
  • 1
    @BrunoLeFloch “One optional space” is the terminology in the TeXbook.
    – egreg
    Sep 26, 2017 at 22:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .