# Why does TeX have so many category codes?

When I look at the long list of category codes, I simply don’t understand why they are all needed. For instance:

• , ^, _, and & could have easily been defined as active characters instead. • Speaking of &, why waste such a common symbol on tab alignment? Why not use something like \nc for “new column”? • Remember that TeX was originally developed by Knuth for his own use in preparing "The Art of Computer Programing". The computer system on which the initial version was created (SAIL on a DECSystem 10) had a nonstandard keyboard, which contained a circled times (familiarly called "splat") that was the original key used as a tab. This was changed for TeX82 to an ascii character that was otherwise unused, the &, For some other related ancient history, see tex.stackexchange.com/a/503994 . – barbara beeton Dec 20 '20 at 20:12 • Regarding category codes vs. active characters, it's quite likely that category coded characters are processed more quickly than active characters, and are also much less likely to be redefined. Again referring to "initial conditions", the available memory and processing speed were also much different than they are today. Knuth took extra care to minimize both, and that almost certainly was relevant to the design choice. – barbara beeton Dec 20 '20 at 20:17 • "long list" ? there are only 16, that is 4 bits per token. – David Carlisle Dec 20 '20 at 20:58 • Another advantage of distinct catcodes over active characters for ^_& (which are all symbols which appear quite often when writing generic macros) is that this allows you to change their definition in your document without affecting loaded packages which might have used the symbol. If they wer active characters, you couldn't give them another meaning without breaking everything. – Marcel Krüger Dec 20 '20 at 20:59
• According to errorlog.tex, a preliminary form of active characters was introduced on 1980-01-25, when category codes were already an established standard. Up to that point, the category codes were just 13 (from 0 to 12), so the active character got number 13. Catcode 14 was introduced later (% used to be given category code 5) and catcode 15 was added to fill the set. – egreg Dec 20 '20 at 22:23

Of course not all characters with special category codes can be replaced with active characters.

Category codes are assigned during the tokenization phase and then are permanently attached to a character token. However, a character with category code 0 never becomes a character token, because it just triggers the mechanism for forming a control sequence. Similarly it's impossible to have character tokens with category code 5 (end-of-line), 9 (ignored), 14 (comment), 15 (invalid).

Codes 0, 5, 9, 14 and 15 all trigger special actions. Historical note: in TeX78 the category code of % was 5 (which didn't allow for comments as we're used to now).

The distinction between category codes 11 and 12 is essential during the formation of control sequences: \a is a control word (and spaces are ignored afterwards), while \? is a control symbol (and spaces are not ignored).

Codes 1 and 2 are likewise essential for macro definitions and argument grabbing (and grouping). Code 10 is fundamental for normalizing spaces and for code indentation. Code 14 corresponds to comments and is likewise important and code 6 is too, for denoting parameters in macro definitions (and in alignments).

Let's now examine codes 3, 4, 7, 8. There could be primitives for them, say \mathshift, \alignment, \superscript and \subscripts. Actually it would be possible to define them by

\let\mathshift=\let\alignment=& \let\superscript=^ \let\subscript=_  and indeed plain TeX and LaTeX have \sp and \sb defined in exactly the same way. Why not using active characters? Well, active characters were introduced in 1980, well after the business with category codes had started. TeX78 only had 13 codes, so active characters got the next slot. Comments as we now know them got slot 14 and code 15 was added to cope with strange characters such as ASCII 127 (which was used with punch cards to delete the previous byte in case of errors so as not to waste a card). But the real reason is that active characters are like macros (more precisely, control sequences) and can be assigned a new meaning at any time. If you say \def\b{foo}\def\a{\b}\def\b{baz}, then a call of \a will produce baz, not foo. Similarly, something like \def\splat{\hbox{\otimes$}}  with $ an active character would not work in case one does \def${foo} afterwards. One would need a primitive \mathshift in place of $ for doing a definition such as \splat; but the problem would be just pushed forward: what if somebody feels the compelling necessity to redefine \mathshift?

Moreover, # cannot be implemented as an active character without changing the core of TeX. Maybe it can be done for &, but a primitive for the initial meaning would be necessary and the problem would be the same as before.

Paranoia? Not at all. There have been questions on the site by people who used \foreach \number in {1,2,...,10} {...} and complained that something went wrong. Or \foreach \color in {<color list>}{...} and hell broke loose.

• +1: Like reading a TeX history book! (positively) – Dr. Manuel Kuehner Dec 21 '20 at 0:57
• Well, the issue that the meaning of an active character may be changed is the case with all commands, including primitives. You can even redefine \def. There’s an easy solution to this problem: Just don’t redefine fundamental TeX constructions. We have managed to live fine with this rule. I don’t know why you think people would be more likely to change the meanings of \$, ^, _, &. I could also redefine \\ , which would cause an equal amount of havoc for table alignment. – Gaussler Dec 21 '20 at 8:32