3

While parsing input, the catcode 11 (letter) and 12 (other) are different (that the latter cannot appear as a part of a multi-character control sequence).

However, when they're already parsed it doesn't seem that they have a lot of difference. They're both

  • not expandable
  • typeset itself when executed
  • converts to the same thing when used inside \scantokens etc.

Apart from making \ifx return false (or \ifcat) and force users to use \tl_analysis_show:N to figure out what is going on, is there any practical use of that?

(also – if this has any answer at all, why was it that way instead of e.g. both are converted to catcode 12? Just a historical artifact, or there's some reason that it makes more sense?)

5
  • Not for any practical purposes, just curious.
    – user202729
    Dec 16, 2021 at 15:16
  • Also for comparison, (single) catcode newline is already converted to space.
    – user202729
    Dec 16, 2021 at 15:19
  • 2
    the answer is I think "no" Dec 16, 2021 at 16:00
  • Once tokenized, a token carries its catcode with it forever. It is an intrinsic part of the token's identity. This becomes a vital feature when catcodes are changed, as it allows code defined prior to the catcode change to behave as expected, even when executed after the catcode change. Dec 16, 2021 at 16:12

2 Answers 2

1

A control sequence can be of two types:

  • a control symbol, or
  • a control word.

In order to make TeX to tokenize a control sequence, you need to type the escape character (usually the backslash, but any character of category code 0 would do); at this point TeX will scan in the following way:

  • if the following character has category code different from 11, the control sequence is a control symbol and its name consists of that single character;

  • if the following character has category code 11, TeX continues to scan until finding a character that hasn't category code 11; the control sequence is a control word whose name consists of the run of characters found in the process.

In case of a control symbol scanning resumes normally; in case of a control word, TeX enters its scanning state “skipping blanks”.

I'd say that the distinction between category code 11 and 12 is very important.

After tokenization the distinction is somewhat blurred; for instance, in hyphenation category code 11 and 12 are accepted to form a word for hyphenation purposes.

On the other hand, \lowercase and \uppercase keep the category code associated to a character, and in case you want to correctly parse something that can come from anywhere, it's perhaps better that category codes are kept. Or not, as the example with \ifx might show. That depends on a decision by Knuth and code can be very dependent on it.

1
  • 3
    This answers the wrong question though, that's input characters looking up current catcode. The question is about the catcode value of character tokens and would it make any difference if 11 and 12 were all normalised to 12 while tokenizing, other than making it easier to test value of \jobname with \ifx. That is similar to all catcode 10 characters being normalised to a normal space token, Dec 16, 2021 at 15:58
1

As I indicated in a comment, once tokenized, a token carries its catcode with it forever. It is an intrinsic part of the token's identity. This becomes a vital feature when catcodes are changed, as it allows code defined prior to the catcode change to behave as expected, even when executed after the catcode change.

Consider the following code:

\documentclass{article}
\begin{document}
\def\z{\Q1}
\def\Q{\textit{Q}}
\expandafter\def\csname Q1\endcsname{\textit{Q1}}

\catcode`1=11 
\def\zz{\Q1}

\z

\zz
\end{document}

To ignore the catcodes would imply that the definitions of \z and \zz are identical. Yet, because of catcodes, this is not so. In the definition of \z, the 1 has a catcode of 12, while in the definition of \zz, the 1 has a catcode of 11.

Despite the fact that the catcode of 1 has been changed along the way to 11, the macro \z, even when executed after the catcode change, still behaves as it did when it was defined (with its 1 possessing a catcode of 12).

Thus the result of \z and \zz are not identical:

enter image description here

If this did not behave in this fashion, the whole TeX construct would collapse. The catcodes in force at the time of tokenization must remain in force for the duration of their use. (while this may not seem perfectly clear considering catcodes 11 and 12, consider the effect when the catcodes of \, { and } are changed---we rely on pre-existing code to carry over the catcodes of 0,1, and 2 in their definitions in order to execute properly)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .