57

I am not asking how the syntax of TeX works. I know perfectly well that only letters are allowed in command names, except for single-character names and constructions using \csname. I’m just asking: Why did Knuth make this peculiar choice? In almost all other programming languages, I can call my commands f1 and f2 and my variables var1 and var2. In TeX, I cannot. Why? What was the intended benefit of this choice?

I know that if numbers had been allowed in command names, the syntax would have had to be adjusted in some places, e.g. I would have to write \kern 5pt rather than \kern5pt. But hey, I actually prefer the former to the latter, so this is hardly a dealbreaker.

1
  • Comments are not for extended discussion; this conversation has been moved to chat.
    – Joseph Wright
    Commented May 25, 2021 at 12:45

8 Answers 8

56
+500

We can give a surprisingly specific answer: the PDP-10 had 36-bit words.


Firstly, note that TeX does allow numbers in "commands", in two senses. TeX's control sequences are of two kinds:

  • A backslash followed by a sequence of letters, like \TeX (the control sequence ends when a non-letter is encountered), or
  • A backslash followed by a single non-letter, like \, or \! or \\ (the control sequence ends after that nonletter).

While many people here have pointed out that by changing the definition of “letter” (assigning the characters 0–9 to category code 11) one can make the digits 0–9 also fall in the first kind of control sequence, note also that even with the default catcodes, TeX allows the ten control sequences \0 \1 ... \9 (of the "single non-letter" kind).

See for instance the source code of The TeXbook (search for The following ``rulers'' have been typeset…), where Knuth defines the macros \1, \2, \3, \4 and \8, and again near Inserting spaces according to the table where he defines \0, \1, \2, \3 and uses them like Ord\2Bin\2Ord\3Rel….

(So if the syntax were to be changed as you suggest, then either \2Bin would no longer mean the control sequence "\2" followed by "Bin", or some special additional rules would have to be added e.g. a control sequence containing digits should not start with a digit, etc. And you mention in the question not minding writing \kern 5pt, don't forget the very common patterns like \fam0 or \box255 which would all have to be rewritten.)

Anyway, to answer your question, you must remember that the initial design of TeX was done as follows:

  1. Knuth looked at a (printed) page of TAOCP Vol 2 (typesetting the second edition of which was the original goal of the whole project).
  2. He thought about what he would like to type, in order to get that typeset output (and which he could also implement easily).
  3. He wrote the program to make it happen.

So the syntax of TeX is based more or less on Knuth's personal preferences for input conventions during those early months of 1977, alongside constraints about what was easy/possible to program on the SAIL system available to him at the time. It has been pointed out in the comments that Knuth seems to prefer typing the short form without spaces (it makes sense for him, as he writes his books with pencil on paper, and uses a computer only to type it up at the end). This is reason enough (see the example from texbook.tex above), but let's also consider the system constraints of the time.

As far as control sequences go, the very first "Preliminary preliminary description of TEX" [sic] did not even have backslashes — see TEXDR.AFT from May 1977 (reprinted as Chapter 24 of the collection Digital Typography). At this point, there were just "keywords", as in eqn. But very soon (see TEX.ONE from July 1977, reprinted as Chapter 25 of Digital Typography), he had adopted the current approach of using an escape character (backslash by default) for control sequences. All this was before a single line of code had been written.

At this point, he was under the impression that control sequences would not be important / much used. He just needed something that would be easy and efficient to implement, to fit on the machine. He implemented something that was "good enough": Chapter 11 of the SAIL TeX (aka TeX78) manual (included in TeX and METAFONT: New Directions in Typesetting), the precursor to The TeXbook, had all these rules (not applicable to current TeX!):

from Chapter 11 of SAIL TeX78 manual

The main point here is "This fact allows TeX to handle control sequences quite efficiently; and TeX's usefulness is not seriously affected".

We can figure out the reason for these rules; how it allowed TeX to handle control sequences efficiently. The SAIL language ran on the computers used at Stanford AI Lab ("SAIL"), which were PDP-10 (aka DECsystem-10) machines, that had 36-bit words. That is, the machine's native integer type could store any number in [0 … 2^36).

Note here that 2*26^7 < 2^36 < 26^8. That is, with an alphabet of 26 English letters, sequences of up to 7 letters could be made to fit in a word (also distinguishing case of the first letter), while 8 or more letters would not fit. If the 10 digits had also been included in this alphabet, then the alphabet would have had size (26+10)=36, and as 36^7 > 2^36, even sequences of length 7 would not have fit; one would have to limit distinct control sequences to length 6, giving up one whole letter's worth of length just for the small benefit of names like \some14u or whatever.

More precisely, the efficient implementation was done in the following way (see the source code, TEXSYN.SAI):

Control sequences, some of which are predeclared, are recorded in a hash table, with an associated table of their equivalent meanings. Linear probing (e.g., Algorithm 6.4L in ACP) is used to access this table […] packed representations of longer control seqences, using six bits for the first letter (in order to distinguish upper and lower case) and five bits for each remaining letter, left justified in the word.

Even at the time of printing of this manual, these restrictions had started to be relaxed: the final Appendix X "Recent extensions to TeX" (starts with "Stop the presses! The following features were added to TeX just before this manual was printed") ends with (the very last words of that TeX78 manual): "Control sequences of any length are now remembered in full; the seven-letter truncation mentioned in Chapter 2 no longer happens".

When TeX was rewritten in Pascal (WEB) during 1980–1982, many of these restrictions were taken away, but the system had been in wide usage by then and the syntax had more or less converged; there was still no perceived need for allowing control sequences mixing letters and digits, and breaking existing usage of control sequences like \0 and \1.

He touches on this at around 42:00 in this 1982 video (part of a series about the internal details of TeX82):

In the SAIL version of TeX… the implementation of strings was quite inefficient: we didn’t want to use SAIL strings for control sequences. Instead we had to go into our dynamic memory and take away valuable space there for the names of control sequences. My first original design of TeX if anybody remembers way back in 1977, I had real strange restrictions that control sequence had to consist of at most five [he means seven —S] letters, and upper- and lower-case were not distinguished after the first one, and things like this was all so that I could keep a separate table of my control sequences. I had the idea at first that hardly anybody would be defining new control sequences.

[Audience laughter]

I didn’t realize that macros were going to be very powerful at first... gradually we found that people actually want control sequences so we've made room for them.

1
  • 11
    Thank you for your meticulous archaeology! The TeX I first learned was indeed TeX78, but I've blessedly forgotten the lack of distinction between upper- and lowercase, and the length limit. This answer is definitive. Commented May 25, 2021 at 16:38
35

Of course, the actual reason why Knuth did this can only be answered by Knuth himself, but we can take a guess based on things we do know:

  1. TeX's rules for determining control sequence names are based exclusively on catcodes which makes them easier to describe, slightly faster to implement and more customizable.

    Since TeX already has 16 different catcodes, digits couldn't have gotten their own catcode without adding a fifth bit to the catcode field in tokens. (Which would be a waste since TeX doesn't need 32 catcodes) So it could only have been implemented in two ways: Allow other tokens in command names or making digits into letters.

    Allowing other tokens in command names would be a real pain since you would often accidentally add punctuation characters to your command names, so the more serious version is to make digits letters. But looking at the texbook source or WEB files written by Knuth indicate that he seems to like the existence of \0 to \9 as short names for control sequences which don't need to be terminated with a space.

  2. Similar to the previous point there also seems to be a much more fundamental reason: You write

    But hey, I actually prefer the former to the latter, so this is hardly a dealbreaker.

    but all code from Knuth seems to indicate that he feels the opposite: He almost always seems to prefer the shorter form without the space. So while allowing digits in control names might not have many downsides for you, it seems to be incompatible with Knuth's preferred style.

9
  • 21
    Given the very limited memory available when TeX was created, simplicity was a paramount consideration. Allowing digits to be used for more than one purpose would have required much more code, for very little advantage. Commented May 23, 2021 at 16:02
  • 5
    In effect, the concept of catcodes means the amount of code needed for lexical analysis of the input is tiny, but with the tradeoff of losing some "nice to have" functionality like variable names containing digits. And efficiency was critical - I can remember running TeX on the original IBM PC where for a document with a fair amount of math, the processing speed was typically one minute per page of output.
    – alephzero
    Commented May 24, 2021 at 3:07
  • 6
    @alephzero -- In its initial Sail implementation, five minutes per page was not unusual. However, the quality obtained far outpaced what was typical at that time with phototypesetting, never mind the ability to produce math, with input in a language that was already familiar to mathematicians (who were at the same time losing skilled secretarial assistance). We've been spoiled by the consequences of Moore's Law. Commented May 24, 2021 at 15:05
  • 2
    @alephzero but... come on, lexical analysis? That ought to be by far the simplest part of the compilation pipeline, and the amount of work done there utterly negligible compared to the intricacies of paragraph layout and maths typesetting. Surely it would have been no problem to toss in one extra bit for the catcodes. Commented May 24, 2021 at 22:35
  • 3
    @leftaroundabout er latex is written in tex that's like saying someone writing a C program should start by forking the C compiler. Not to mention that by the time Leslie had stopped working on latex, it still needed to fit into a 540K PC running emtex. These bit packing constraints are everywhere in tex: that is why you can only have 16 math families or why there are not more math classes or ... and the question is not, how would it be done now, it's why was it done that way then and I was just pointing out that using 5 bits would be a far from trivial change, Commented May 25, 2021 at 11:23
24

Given the very limited memory available when TeX was created, simplicity was a paramount consideration. Allowing digits to be used for more than one purpose would have required much more code, for very little advantage.

(This was originally posted as a comment, but since there seems to be some agreement, I'll extend it.)

Tasks that would need to be accommodated in some other way if digits were \catcoded to be letters:

  • later changes to any \catcodes;
  • any arithmetic operations, including incrementing counters;
  • specifying or resetting any dimensions, including those used to reposition glyphs or blocks of text or other objects.

There are no doubt more, but these alone would destroy the simplicity of recognizing digits for the uses for which they are most appropriate.

Addendum:

It must be remembered that the purpose for creating TeX was to make it possible to typeset The Art of Computer Programming (TAOCP) in a manner consistent with and to the high quality standards exhibited by the original volumes of the series, set most competently via Monotype. The design and implementation of TeX achieves that goal. It was only later that Knuth was persuaded that making the software available to others, and in doing so, he accepted suggestions that extended TeX's capabilities without compromising its original purpose or efficiency. The last (and most extensive) such extension was the inclusion of natively accented letters (essentially Latin-1) in 1989, marking version 3.0.

Knuth makes it clear on his TeX web page that additional extensions will not be made, and, although such extensions may be made by anyone who so chooses, the resulting program should be named something other than "TeX"; that name remains reserved for Knuth's own work.

2
  • 7
    I just think there is a tendency that whenever someone asks, “Why is this extremely elementary feature that all other programming languages have not in TeX”, the answer you usually get is that this is veeeery difficult to implement. Hey, C is older than TeX, and it doesn’t have this problem. Yes, probably many thousands of lines of code would have to be changed, but are we supposed to pretend that this is not doable, in a manageable amount of time? Certainly, it’s a smaller task than, say, adding Lua to TeX. ;-)
    – Gaussler
    Commented May 24, 2021 at 10:35
  • 3
    @Gaussler It's not undoable, of course. There is just not enough people with enough free time to devote to a slight change in syntax (that in my opinion would not bring that much of a benefit) Commented May 24, 2021 at 17:13
12

Since the OP seems interested, nonetheless, in how this might be done, I have included a small MWE. Here, I introduce the \z pseudo-environment that will temporarily operate on digits as cat-12, in order to accomplish otherwise painstaking tasks.

I show first how things like \hspace and \numexpr would be used without the temporary cat-12 digits, and then show again, with the use of \z...\endz how they could be alternately employed. After that, I show how \z...\endz can be used for parameterization of #1, #2, etc., as used inside \defs.

To clarify for those wondering, \z does operate within a group, and there collects (and thus tokenizes with catcodes) the tokens of the group (with cat-12 digits) into the \cytoks token list. Then, at the end of the environment, instead of performing the customary typesetting of \the\cytoks, I instead delay this function until fully exited from the environment group. Thus, things like \def are collected inside the group, but executed outside (and after) the group, so no \global is required.

\documentclass{article}
\usepackage{tokcycle}
\xtokcycleenvironment\z
{\whennotprocessingparameter##1{\addcytoks{##1}}}
{\processtoks{##1}}
{\addcytoks{##1}}
{\addcytoks{##1}}
{\catcode`\1=12 \catcode`\2=12 \catcode`\3=12 
 \catcode`\4=12 \catcode`\5=12 \catcode`\6=12 
 \catcode`\7=12 \catcode`\8=12 \catcode`\9=12 
 \catcode`\0=12}
{\cytoks\expandafter{\expandafter\cytoks\expandafter{\the\cytoks}}%
  \tcafterenv{\the\cytoks}}
\expandafter\def\csname No1\endcsname{1}
\expandafter\def\csname No2\endcsname{2}
\expandafter\def\csname No3\endcsname{3}
\expandafter\def\csname No4\endcsname{4}
\expandafter\def\csname No5\endcsname{5}
\expandafter\def\csname No6\endcsname{6}
\expandafter\def\csname No7\endcsname{7}
\expandafter\def\csname No8\endcsname{8}
\expandafter\def\csname No9\endcsname{9}
\expandafter\def\csname No0\endcsname{0}
\AtBeginDocument{\z
\catcode`\1=11 
\catcode`\2=11
\catcode`\3=11
\catcode`\4=11
\catcode`\5=11
\catcode`\6=11
\catcode`\7=11
\catcode`\8=11
\catcode`\9=11
\catcode`\0=11
\endz}
\begin{document}

x\hspace{\No1\No2 pt}x
\the\numexpr \No1 + \No2 

%\hspace{12 pt} THIS WILL BREAK, PROVING DIGIT CATCODES =11
x\z\hspace{12pt}\endz x
\z\the\numexpr 1+2 \endz

\hrulefill

\z\def\mymacro#1{x\hspace{#12pt}x \the\numexpr#1 +2}\endz
\mymacro{\No2}

\mymacro{\No3}

\z\def\mymacro#1#2{(#1--#2)}\endz
\mymacro{1}{23}
\end{document}

enter image description here

10

In The TeXbook I could not find a reason why Knuth made the choice to allow only letters by default. However, he did acknowledge the potential need for more flexible control sequences, and pointed out this can be achieved with \catcode.

See the following exercise on page 11, with the solution on page 301 (lines 911-921 in the TeXbook source):

enter image description here

enter image description here

4
  • 1
    That sounds like a bug in the TeX book then. How can it be 256^2 if you need at least 1 char with catcode 1 to start it? Commented May 25, 2021 at 18:13
  • 1
    @FrankMittelbach Interesting point. :) Not strictly a bug maybe, as the idea I guess would be to use different escape characters depending on which two characters are in the control sequence you want to refer to. (Like: if you've defined a control sequence made of the two "letters" \ and !, then you can still refer to this specific one using a different escape character say ;, like ;\!. Then change catcodes again when you want to use a control sequence containing ; as a "letter".) Commented May 25, 2021 at 19:58
  • @FrankMittelbach In my opinion, the bug in the TeXbook might be that by using the word "possible" without clarification the question is not formulated with sufficient precision. The amount of length 3 possible at a single point in time indeed is something like (256 - [amount of characters not of catcode 11])^2. At least one character must be of catcode 0(escape). So in any case at least one character is not of catcode 11. So the largest amount of 3 that is possible at a single point in time is (256 -1)^2 = (255)^2. But if "possible" also subsumes catcode-changes/several points in time... Commented May 26, 2021 at 0:29
  • 1
    @ShreevatsaR you and Ulrich are right, the question would allow for the escape char to change back and forth (and I should have written catcode 0) Commented May 26, 2021 at 13:13
8

Whereas my other answer takes the approach of actually changing the catcodes of the digits 0-9, we see that it introduces many difficulties that may not be worth the effort.

In contrast, this approach changes no catcodes, but instead uses an active character (here taken as / since it looks kind of like \, and one can use \slash as a text replacement) to act almost like a catcode 0 token, but to include the digits 0-9 when both defining as well as invoking "macros". If you prefer to sacrifice a different token than /, just replace every instance of / in the MWE with the desired active character. The limitation of this approach, as we shall see, is that the process of tokenizing alphanumeric macro names is not expandable.

The idea came to me from my tokcycle package, which is confronted with the challenge of parsing the tokens of an input stream in a single pass. Two difficult things to do are differentiating a { from a \bgroup and differentiating a space from an implicit space. The way tokcycle accomplishes this is to take the \string of the token. In the end, tokcycle does this to every cat-0 control sequence in the input stream as a matter of course. The problem, then, for those tokens that are now control sequence \strings, is reconstituting the macro from which the \string originated, since it was a string and not a macro that was initially absorbed.

tokcycle uses a macro \csmk which performs a faux tokenization of the string (meaning it uses a macro rather than the TeX mouth to tokenize the input stream), but using the rules of control-sequence name construction. I decided, here, to start with \csmk, rename it to \xsmk, and adjust it to include digits 0-9 as well as cat-11 tokens in the process of faux tokenization.

I then place a wrapper around this macro that uses an active character / both to invoke macros defined with this faux tokenization as well as to define them.

The syntax is /def/<alphanumeric string>{...} to define a macro that is invoked by way of /<alphanumeric string>. It will even work with delimited arguments. In the MWE, I show examples of these.

Then, purely for demonstration, I revert two definitions used in \xsmk so that it behaves in a fashion similar to the original \csmk (except operating on cat-11 tokens, instead of cat-12 string tokens). I then run the same input through and you will see how the faux tokenization excludes digits from the macro names.

In summary then, this approach enables the definition and non-expandable invocation of alphanumeric macro names (which are internally stored as \csname constructions) with the use of an active token / that functions as if it were a catcode-0 \.

\documentclass{article}
\usepackage{tokcycle}

% FAUX TOKENIZATION BASED ON tokcycle MACRO \csmk
\makeatletter
\def\xsmk#1{\def\xsaftermk{#1}\toks0{}\@xsmkA}
\def\@xsmkA{\futurelet\@tmp\@xsmkB}
\def\@xsmkB{\tctestifx{\@tmp\tc@sptoken}%
  {\toks0{ }\expandafter\@xsmkF\tc@absorbSpace}{\@xsmkCA}}
\def\@xsmkCA#1{\tc@addtoks0{#1}\tctestifnum{\numberCatcode#1=11}%
  {\futurelet\@tmp\@xsmkD}{\@xsmkF}}
\def\@xsmkC#1{\tctestifnum{\numberCatcode#1=11}
  {\tc@addtoks0{#1}\futurelet\@tmp\@xsmkD}{\@xsmkE#1}}
\def\@xsmkD{\tctestifCatnx A\@tmp\@xsmkC\@xsmkE}
\def\@xsmkE{\tctestifx{\@tmp\tc@sptoken}%
  {\expandafter\@xsmkF\tc@absorbSpace}{\@xsmkF}}
\def\@xsmkF{\tc@defx\thecs{\csname\the\toks0\endcsname}\xsaftermk}

\def\tc@addtoks#1#2{\toks#1\expandafter{\the\toks#1 #2}}
\makeatother

% ADJUST TO INCLUDE cat-11 LETTERS AS WELL AS DIGITS 0-9
\def\numberCatcode#1{%
  \tctestifcatnx 1#1
    {\tctestifx{#11}{11}
    {\tctestifx{#12}{11}
    {\tctestifx{#13}{11}
    {\tctestifx{#14}{11}
    {\tctestifx{#15}{11}
    {\tctestifx{#16}{11}
    {\tctestifx{#17}{11}
    {\tctestifx{#18}{11}
    {\tctestifx{#19}{11}
    {\tctestifx{#10}{11}
    {\number\catcode`#1}}}}}}}}}}%
  }{\number\catcode`#1}%
}
\long\def\tctestifCatnx#1#2#3#4{%
  \tctestifcatnx#1#2{#3}%
  {\tctestifx{#21}{#3}
  {\tctestifx{#22}{#3}
  {\tctestifx{#23}{#3}
  {\tctestifx{#24}{#3}
  {\tctestifx{#25}{#3}
  {\tctestifx{#26}{#3}
  {\tctestifx{#27}{#3}
  {\tctestifx{#28}{#3}
  {\tctestifx{#29}{#3}
  {\tctestifx{#20}{#3}{#4}}}}}}}}}}}%
}

% ACTIVE CHARACTER WRAPPER
\catcode`\/=\active
\def/{\xsmk{\parsecs}}
\def\parsecs{\expandafter\tctestifx\expandafter{\thecs\def}{\Qdef}{\thecs}}
\def\Qdef/{\xsmk{\Qdefhelp}}
\def\Qdefhelp{\expandafter\def\csname\the\toks0\endcsname}

% DEFINE SOME WORKING MACROS WITH \def AND /def
\def\1{[1]}
\def\a{[a]}
\def\aa{[aa]}
/def/1a{[1a]}
/def/a3{[a3]}
/def/aa1{[aa1]}
/def/a7a{[a7a]}
/def/b3-#1-{[b3:#1]}% MACRO /b3 WITH DELIMITED -#1- ARGUMENT
\def\b{[b]}

\begin{document}
\def\doit{
/a
/today ,
/a3.
/a3 .
/aa1.
/aa1{}
/1a.
/a7a
/b3-xyz-

\tctestifCatnx AaTF
\tctestifCatnx A.TF
\tctestifCatnx A1TF

\numberCatcode a,
\numberCatcode .,
\numberCatcode 1
}

\doit

RECOVER ORIGINAL TOKENIZATION
\def\numberCatcode{\number\catcode`}
\let\tctestifCatnx\tctestifcatnx

\doit
\end{document}

enter image description here

In case one was wondering in terms of how to adapt \xsmk to other uses, the syntax is \xsmk{<after-code>}<input stream>. With such an invocation, a macro name is constructed from the leading tokens of the input stream, using the rules provided, and stored in \thecs. Then, <after-code> is executed.

6

For similar aesthetic reasons that you want to be able to write \banana5, Knuth apparently wanted to be able to write \box4 and \dimen12. Making numbers part of control words would have required writing \box 4 and \dimen 12 to achieve the same effect, and when formatting something like \dimen 12=4pt\relax this could have been broken across lines at the space.

In particular for writing the TeXbook, this would have caused extra input/formatting work and pain and would have led to awkward reading since plain TeX works a lot using registers. Being able to use pretend-names like \dimen12 helps to keep it readable, exactly for the same reasons that you'd like something like \banana5 to be a single identifier.

6
  • Alternatively, as @egreg pointed out, \banana{5}, \dimen{12}, and \box{4} would have been a much better syntax.
    – Gaussler
    Commented May 25, 2021 at 15:55
  • 2
    @Gaussler -- The problem with your proposal is that it requires a lot more typing (at least for input of the TeXbook), and also, in many cases, would require definitions to include an extra argument, a situation that traps a lot of TeX newbies with quite incomprehensible errors. Commented May 25, 2021 at 16:28
  • @barbarabeeton, honestly, for anyone who learned TeX in the 21st century, I think syntax like \kern5pt is what is really causing confusion. And do you seriously think that a few braces here and there count as “a lot more typing”? If Knuth could spend 10 years on developing TeX, I’m sure he could also add a few braces here and there.
    – Gaussler
    Commented May 30, 2021 at 11:51
  • @Gaussler -- Okay. Forget the "lot more typing". With digits allowed in control words and requiring braces in other environments, there are now three distinct syntactic uses for digits, and getting braces around the correct ones requires different thinking patterns. Most important, it makes debugging a lot harder, and for someone in whose profession debugging is central, that makes a difference. You're welcome to use the TeX code and develop a variant that you like. (Just don't call it TeX.) Commented May 30, 2021 at 15:13
  • @barbarabeeton Good. But I, for one, would have liked to be able to make a control sequence \C1 for the category \mathcal{C}_1. I could, of course, call it \Cone, but this one is already taken for the cone construction, and the name is far less intuitive. Don’t get me started on \projonethree for the projection \pi_{13}. Numbers not being allowed in control sequences is annoying me at least five times a day. Luckily, I have my stricttex package which takes care of it, but which unfortunately requires LuaTeX.
    – Gaussler
    Commented May 30, 2021 at 15:40
5

Multiletter-control-sequences in the .tex-input intended to yield a single ⟨control-word-token⟩ during tokenization are to consist of a single character of category 0(escape) trailed by a (non-empty) sequence of characters of category 11(letter).

Characters that represent decimal digits (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) usually are assigned category 12(other). Therefore digits usually are not considered parts of names of ⟨control-word-tokens⟩ that come into being directly via tokenizing .tex-input. (⟨control-word-tokens⟩ coming into being as the result of the toplevel-expansion of \csname...\endcsname is a different story.)

Why at all having TeX distinguish a set of category 11(letter)-characters from a set of category 12(other)-characters instead of having TeX just "look" at the union of these two sets?

TeXbook, Appendix B: Basic Control Sequences says:

When INITEX begins, category 12 (other) has been assigned to all 256 possible characters, except that the 52 letters A...Z and a...z are category 11 (letter), and a few other assignments equivalent to the following have been made:

\catcode `\\ =0    \catcode`\ =10    \catcode `\% =14
\catcode`\^^@=9    \catcode`\^^M=5    \catcode`\^^?=15

A double-dangerous-bend-paragraph of Chapter 7, How TeX reads what you type, says the same. Chapter 7 also says that token-producing-"operations" like \string, \number, \romannumeral produce character-tokens of category 12(other). The same is for \the in many situations.

To me it seems that category 12(other) is seen as the "standard".

To me it seems category 11(letter) was introduced for denoting (besides characters of category 0-10 and 13-15) another set of "non-standard"-characters. A set of "non-standard"-characters whose properties deviate from the properties of all other characters in the aspect that only these characters can be used within .tex-input for denoting names of ⟨control-word-tokens⟩.

I don't know why Professor Knuth decided for introducing a category for explicitly declaring a set of characters that can be used for denoting names of ⟨control-word-tokens⟩.

But I think having such a set of characters defined can be very handy. E.g., for more easily judging whether a sequence of characters forms the name of a ⟨control-word-token⟩ when looking at a piece of TeX-code with one's eyes.

However, I think that the fact that digit-characters by default are not elements of this set and therefore usually are not considered parts of names of ⟨control-word-tokens⟩ that come into being directly via tokenizing .tex-input is not due to a strong desire to specifically prevent/forbid/dis-allow the usage of digits in names of ⟨control-word-tokens⟩.

I suppose digit-characters by default usually not being considered parts of names of ⟨control-word-tokens⟩ that come into being directly via tokenizing .tex-input is just a side-effect taken for acceptable.

I suppose the main reason for assigning category 12(other) to digits (and this way—as a "side-effect"—excluding digits from that set of characters that can be used for denoting names of ⟨control-word-tokens⟩) is:

According to TeXbook, Chapter 24: Summary of Vertical Mode ⟨digit⟩-quantities are explicit character-tokens of catcode 12(other)012|112|212|312|412|512|612|712|812|912 (not catcode 11(letter)! ).

E.g., within macro-definitions a macro-argument is denoted by a category 6(parameter)-character-token that is trailed by such a (category-12-)⟨digit⟩-quantity in range 1..9:

\catcode`\1=11
\def\foo#1{this is foo}
\bye

yields an error-message "! Parameters must be numbered consecutively."

Assigning category 12(other) to digits makes it possible to use them as ⟨digit⟩-quantities, e.g, for denoting macro-arguments or as components of ⟨(8-bit) number⟩-quantities. (⟨8-bit number⟩-quantities are a special case of ⟨number⟩-quantities and they are used with things like \catcode).

Using characters/digits 0 ,1, ..., 9 as ⟨digit⟩-quantities is not possible if they have category 11(letter).

So a sort of reason why you usually cannot use 0, 1, ..., 9 in command names is that assigning them category 11(letter), which is the only way of making it possible to use them in command names, excludes using them as ⟨digit⟩-quantities, e.g., in macro-arguments #1, #2, etc, or as components of ⟨number⟩-quantities as ⟨digit⟩-quantities by design must be of category 12(other). Token-producing routines like \the⟨count-register⟩ and \number and \string all also produce character-tokens of category 12(other). (Exception: Spaces produced by \string are of category 10.)

I said "a sort of reason" because, given how TeX is implemented, this just explains why switching digits to category 11(letter)—and this way allowing them to be components of the names of control-word-tokens—might cause problems/is not a good idea and why this this is not done by default. But this is not an explanation why TeX is implemented in a way where switching digits to category 11(letter)—and this way allowing them to be components of the names of control-word-tokens—can lead to problems.

Probably it would have been nice to allow ⟨digit⟩-quantities to be not only of category 12(other) but to be of category 11 also and to assign digits to category 11(letter) by default. But that ship has sailed—e.g., when denoting numbers of registers people in their .tex-input-files omit the space between \count/\box/\toks/\skip and the digit-sequence that denotes the number.


As under normal category code régime you cannot obtain the ⟨control-word-tokens⟩ \var1 and \var2 directly by having TeX read and tokenize .tex-input and as correct invocation of \csname..\endcsname in combination with \expandafter sometimes seems cumbersome, I offer a macro \CsNameToCsToken to create, e.g., the ⟨control-word-token⟩ \var1 from the ⟨character-token⟩-sequence v11a11r11112 or the ⟨control-word-token⟩ \var2 from the ⟨character-token⟩-sequence v11a11r11212.

The phrase "create the ⟨control-word-token⟩" here doesn't focus that much on defining a macro. The focus is more on the stage of expanding the macro \CsNameToCsToken and hereby replacing ⟨character-tokens⟩ nested between curly-brace-tokens ({1 and }2) by a ⟨control-word-token⟩—be that ⟨control-word-token⟩ defined or not (yet) defined.

\CsNameToCsToken does things like \csname..\endcsname for you and is intended to prevent you from the need of inserting \expandafter and the like in the right places which can be tricky or—depending on your attitude and level of knowledge—painful. It is not a means for directly writing \var1 or \var2 within the .tex-input.

Syntax:

\CsNameToCsToken⟨stuff not in braces⟩{⟨NameOfCs⟩}

⟨stuff not in braces⟩\NameOfCs

(⟨stuff not in braces⟩ may be empty.)

(⟨stuff not in braces⟩ is treated as {-delimited argument via TeX's #{-notation.
Due to \romannumeral-expansion the result is obtained by triggering two expansion-steps, e.g., by having two "hits" with \expandafter.)

Examples:

With such a macro you are not bound to specific definition commands:

\CsNameToCsToken{foo}\foo  .

\CsNameToCsToken\global\long\outer\def{foo}\global\long\outer\def\foo  .

\CsNameToCsToken\expandafter{foo}\bar\expandafter\foo\bar  .

\CsNameToCsToken\let{foo}=\bar\let\foo=\bar  .

\CsNameToCsToken\CsNameToCsToken\let{foo}={bar}\CsNameToCsToken\let\foo={bar}\let\foo=\bar  .

\CsNameToCsToken\string{foo}\string\foo  .

\CsNameToCsToken\meaning{foo}\meaning\foo  .

\CsNameToCsToken might be useful not only with plain TeX. It can as well be applied with LaTeX 2ε- and xparse-definition-commands:

\CsNameToCsToken\newcommand{foo}\newcommand\foo  .

\CsNameToCsToken\DeclareRobustCommand{foo}\DeclareRobustCommand\foo  .

\CsNameToCsToken\NewDocumentCommand{foo}...\NewDocumentCommand\foo...  .

Using plain TeX, not LaTeX:

\begingroup
\def\foo#1{#1}%
\catcode`\@=11
\foo{%
  \endgroup
  %%===============================================================================
  %% End \romannumeral-driven expansion safely:
  %%===============================================================================
  \chardef\UD@stopromannumeral=`\^^00
  %%===============================================================================
  %% Obtain control sequence token from name of control sequence token:
  %%===============================================================================
  %% \CsNameToCsToken<stuff not in braces>{NameOfCs}
  %% ->  <stuff not in braces>\NameOfCs
  %% (<stuff not in braces> may be empty.)
  \long\def\CsNameToCsToken#1#{\romannumeral\InnerCsNameToCsToken{#1}}%
  \long\def\InnerCsNameToCsToken#1#2{%
    \expandafter\UD@exchange\expandafter{\csname#2\endcsname}{\UD@stopromannumeral#1}%
  }%
  \long\def\UD@exchange#1#2{#2#1}%
}%


\begingroup\tt
\noindent Define \string\var1:
\hfill\break
\string\CsNameToCsToken\string\def\string{var1\string}\string{This is the current value of var1.\string}
\endgroup
\CsNameToCsToken\def{var1}{This is the current value of var1.}
\hfill\break\null

\begingroup\tt
\noindent Define \string\var2:
\hfill\break
\string\CsNameToCsToken\string\def\string{var2\string}\string{This is the current value of var2.\string}
\endgroup
\CsNameToCsToken\def{var2}{This is the current value of var2.}
\hfill\break\null

\begingroup\tt
\noindent Using \string\var1/var1: \string\CsNameToCsToken\string{var1\string}
\hfill\break\null\hfill\break\null
\endgroup
\noindent \CsNameToCsToken{var1}
\hfill\break\null

\begingroup\tt
\noindent Using \string\var2/var2: \string\CsNameToCsToken\string{var2\string}
\hfill\break\null\hfill\break\null
\endgroup
\noindent \CsNameToCsToken{var2}

\bye

enter image description here

Using LaTeX:

\makeatletter
%%===============================================================================
%% End \romannumeral-driven expansion safely:
%%===============================================================================
\@ifdefinable\UD@stopromannumeral{\chardef\UD@stopromannumeral=`\^^00}%
%%===============================================================================
%% Obtain control sequence token from name of control sequence token:
%%===============================================================================
%% \CsNameToCsToken<stuff not in braces>{NameOfCs}
%% ->  <stuff not in braces>\NameOfCs
%% (<stuff not in braces> may be empty.)
\@ifdefinable\CsNameToCsToken{%
  \long\def\CsNameToCsToken#1#{\romannumeral\InnerCsNameToCsToken{#1}}%
}%
\newcommand\InnerCsNameToCsToken[2]{%
  \expandafter\UD@exchange\expandafter{\csname#2\endcsname}{\UD@stopromannumeral#1}%
}%
\newcommand\UD@exchange[2]{#2#1}%
\makeatother

\documentclass{article}

\begin{document}

\begin{verbatim}
Define \var1:
\CsNameToCsToken\newcommand*{var1}{This is the current value of var1.}
\end{verbatim}

\CsNameToCsToken\newcommand*{var1}{This is the current value of var1.}

\begin{verbatim}
Define \var2:
\CsNameToCsToken\newcommand*{var2}{This is the current value of var2.}
\end{verbatim}

\CsNameToCsToken\newcommand*{var2}{This is the current value of var2.}

\begin{verbatim}
Using \var1/var1: \CsNameToCsToken{var1}
\end{verbatim}
\CsNameToCsToken{var1}

\begin{verbatim}
Using \var2/var2: \CsNameToCsToken{var2}
\end{verbatim}
\CsNameToCsToken{var2}

\end{document}

enter image description here

13
  • I think the argument here is a bit backwards since if Knuth would have wanted to make digits letters, he could just have changed the scanning rules to accommodate this. It wouldn't have been any more complicated to implement the system with all digits being letters. (The only change would have been to replace define zero_token = other_token + "0" with define zero_token = letter_token + "0" in the source code) Commented May 23, 2021 at 20:55
  • @MarcelKrüger Would the change lead to token-producing-operations like \number and \the also producing digits of catcode 11(letter)? If not, then this would lead to the result of \number, which is catcode-12-character-tokens, not being "evaluatable" by \romannumeral or another \number any more - things like \romannumeral\number\number#1 000 for obtaining #1 catcode-12-characters "m"(e.g., with #1=\count11) wouldn't work out. Commented May 23, 2021 at 21:08
  • That's an excellent argument. Even if the \the expansion would create letter catcodes for digits it would lead to very messy rules there. Commented May 24, 2021 at 23:06
  • While there are some good arguments here, they're expressed using LaTeX terminology. LaTeX diverges in a number of ways from Knuthian TeX, is not used by Knuth, and he has voiced strong disagreement with some LaTeX decisions (though not in the area of how digits are used). Thus the arguments here are weakened by association. Still, worth an upvote for the effort. Commented May 26, 2021 at 21:28
  • @barbarabeeton The example exhibiting \CsNameToCsToken is LaTeX, yes, but can easily be adapted to plain TeX. Which other arguments/phrases are LaTeX terminology? Commented May 26, 2021 at 21:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .