What are category codes?

Question

Following on from this question, I'd like to ask a more general question:

What are category codes, and what can I achieve by changing them?

Answer 1

When TeX parses input, it assigns each character read a category code. How TeX subsequently interprets the input then depends on both the character and it's category code. There are 16 category codes that can be set by the programmer, plus one special internal one. The 16 standard ones number from 0 upward. Category code 0 is for escape characters, usually \. The rest are then (with typical examples):

1. Begin group: {
2. End group: }
3. Math shift: $
4. Alignment: &
5. End-of-line
6. Parameter for macros: #
7. Math superscript: ^
8. Math subscript: _
9. Ignored entirely
10. Space
11. Letters: the alphabet.
12. 'Other' character - everything else: ., 1, :, etc.
13. Active character - to be interpreted as control sequences: ~
14. Start-of-comment: %
15. Invalid-in-input: [DEL]

Now when TeX reads input, each character is associated with a category code to generate tokens. So if the input reads

$ 1^{23}_a $

TeX reads:

• A math shift token, and goes into math mode
• A space, which is ignored in math mode
• An 'other' token 1, which is simply typeset here
• A math superscript token, thus meaning that the next item will be superscripted
• A begin-group token,
• The 'other' tokens 2 and 3, which cannot be typeset until the group finishes
• The close-group token }, which allows TeX to typeset the superscript
• A math subscript token, so moving the next item to a subscript position
• The letter a, which with no special meaning is typeset
• A space, again ignored
• A math shift token, and goes back into horizontal mode

Category codes often become important when TeX is deciding on what is and is not a control sequence. With only the alphabet as 'letters', something like

\hello@

is the control sequence \hello followed by the 'other' token @. On the other hand, if I make @ a letter

\catcode`\@=11\relax
\hello@

then TeX will look for a macro called \hello@. This is commonly used in TeX code to isolate 'code' macros from 'user' ones. So you find programming macros such as \@for. Without changing the category code, this is effectively 'hidden'. The idea of this is to 'protect the user from themselves': it's hard to break the code if you cannot even get at it!

There are many effects that can be achieved using category codes. An obvious one is the non-breaking space ~ used throughout the TeX world. This works because ~ has category code 13, and is therefore 'active'. When TeX reads ~, it looks for a definition for ~ in the same way it would for a macro. That's a lot more convenient than using a macro for these cases.

We can use different category codes to make 'private' code areas. For example, plain TeX and LaTeX2e us @ as an extra 'letter', whereas LaTeX3 uses : and _. That effectively isolates internal LaTeX3 code from LaTeX2e, when the two are used together (as at present).

Verbatim material is another area where category codes are vital (if complex!). The reason you can't nest verbatim material inside anything else is that once TeX has assigned category codes it is only partially reversible. Anything which is 'ignored' or 'comment' is thrown away: you can't get it back. (With e-TeX, you can reassign category codes, but anything that is already gone stays 'lost.)

---

(Note for the interested) The 'special' category code is 16, which is used in the \ifcat test, amongst other things. It is assigned to unexpandable control sequences in this situation, so that they do not match anything else other than other unexpandable control sequences.

Answer 2

Rather than defining primitive commands for common tasks such as starting math mode or denoting superscripts and subscripts, Knuth decided to reserve some characters for these purposes. There are also other needs: grouping, denoting the macro parameters and, most important, escaping in order to express commands.

There are sixteen category codes:

 0 = escape
 1 = group start
 2 = group end
 3 = math shift
 4 = alignment tab
 5 = end of line
 6 = parameter
 7 = superscript
 8 = subscript
 9 = ignored character
10 = space
11 = letter
12 = other character
13 = active character
14 = comment
15 = invalid character

Usually there's only one character having categories 0 to 8

\ { } $ & ^^M # ^ _

(^^M denotes the invisible character that TeX puts at the end of all input lines, changing the system dependent one(s) that might be present); uniqueness is not required, but preferred: why should one want to have two different escape characters which would act just in the same way? (See later on.)

Category 10 is the space but also the <TAB> character, that is not distinguishable from a sequence of spaces; category 10 characters are ignored at the start of a line. Category 5 is very special: it's transformed into a space unless it's followed by another category 5 character, when it becomes the command \par (it's the trick that allows to leave a blank line to end a paragraph). In general any sequence of contiguous category 10 characters is reduced to only one and it doesn't matter if they are spaces, tabs or converted end-of-line characters.

All letters have category 11 and punctuation characters such as ?, (, ) and others have category 12; this is for the rule that a command name can be any sequence of letters (better, category 11 characters) or one not 11 category character, preceded by a category 0 character. Category 11 and 12 characters, when not part of a command name may be printed; this is not the case for all other category codes. However a category code 11 or 12 character may also not show up in print, because it's discarded during processing (for example keywords or option to packages, package or file names, ...).

Category 9 and 15 were put into TeX because there are "dangerous" character (ASCII "null" and ASCII "delete") that could be misinterpreted by editors. Actually category 9 has other uses: in LaTeX3 style files the space is assigned category 9, to help programmers in avoiding the dreadful "spurious spaces".

Category 14 is the well-known % that introduces comments and makes TeX ignore everything following it in a line (end of line included).

Category 13 is very special; Plain TeX and the LaTeX kernel use only one active character, namely ~; an active character is treated as if it were a command and must have a definition before it can be used; the LaTeX definition is

\catcode`~=13
\def~{\nobreakspace{}}

so that typing ~ is just the same as writing \nobreakspace{}. Other active characters are also used by the LaTeX "inputenc" package, in such a way that, for instance, ü is translated into \"u.

When we want to typeset verbatim TeX code, many of the special characters are assigned category code 12; but when we type \verb+\xyz+, LaTeX reads \verb and prepares everything for verbatim typesetting and starts a group; the first + is swallowed and is assigned category 2, so that when it finds the second + the group is terminated and all assignments are reverted to the normal ones (including the category 2 assignment to +): it's a bit magic, but it works, provided \verb+\xyz+ doesn't appear in the argument of a command.

This is a problem: when TeX is scanning the argument to a command, it freezes the category codes: when a character enters TeX it is transformed into a pair (category code, character code) which is no more the original character and so the category code assignment can't be modified any more (well, not really, there's \scantokens, but this would require a very long discussion).

The LaTeX commands \makeatletter and \makeatother work by changing the category code of @, which is usually 12; the first one puts it into category 11, so that it can appear in command names, the second one reverts this assignment. But how can

\makeatletter
\newcommand{\xyz}{...\@xyz...}
\makeatother

work? One might expect that when TeX expands \xyz it finds the "illegal" command name \@xyz. This doesn't happen: just as a simple character is transformed into a pair of numbers, when TeX scans it, a command name becomes a symbolic token, an internal representation of the command which is independent of characters and their category codes.

If we assign category code 0 to |, we can type \LaTeX and |LaTeX: they would mean just the same thing. But having different characters sharing the same category code 4 might turn out to be useful for aligning decimal numbers at the decimal separator in a tabular. If we assign . category code 4, we may type a decimal number as 123.456 and LaTeX will interpret it as if it were 123&456, producing two table cells that to the end user appear as one; some trickery in the definition of the table column structure is required, though.

Answer 3

This is not an easy topic and I can only refer you to the TeXBook for more details, but here is a short outline.

Every character that TeX reads from your file has two numbers associated with it. A "character code" and a "category code". TeX does not know glyphs - only numbers - and this is part of its strengths. You can think of font tables as look-up list. If you give it a number TeX will look at the list and print the glyph that happens to be in that position.

The second number is the "category code". This TeX uses it to intelligently parse the input. TeX needs to know for example if a curly left bracket { has appeared in a particular part of the document so that it can look for an ending bracket and so on. Of course this could have been hardwired, but Knuth chose to abstract it, so that any character, provided it has the appropriate category code can be used.

Consider that you may wish to replace curly brackets with square brackets [], this can be achieved by the following simple code:

\catcode `[=1
\catcode `]=2

\def\test[This is a test]
\bye

Try it out without the catcode changes and it will fail with a run-away definition error.

Similarly, the \ backslash character can be redefined for use in verbatim text.

\catcode `[=1
\catcode `]=2
\def\test[This is a test]
\catcode `*=0
 *def*test[This is another test]
 *test
 \bye

In the last example in lieu of \ you can type *. Run both MWE through pdfTeX. Authors don't really need them but they are invaluable for package developers.

Answer 4

There are great answers here already so let me just show a small example for what you can do by changing catcodes. Assume that you have numbers in a table like 12.3 and 8.45 and you want to pad them all to the same width in order to make them line up nicely. (Let's ignore for a moment that there are several ways.) You can do this by adding an invisible 0 after 12.3 and before 8.45. The invisible 0 is provided by \phantom{0} (or \phantom0). Now, using this is somewhat unpleasant to type because it messes up your source code layout. But you could make _ locally active and give it the meaning of \phantom0:

\catcode`_=\active
\def_{\phantom0}

Now you can use 12.3_ and _8.45 in your table. You could have used some other character as well, like ! for example. Since _ is used in math mode for subscripts this only works when you need no subscripts where the above definition is active.