7

Which tokens does TeX not use as undelimited arguments (unless nested between an explicit character token of category code 1 and an explicit character token of category code 2)?

In the last but one dangerous bend paragraph before exercise 20.4 of the TeXbook you find the sentence:

After you have said ‘\def\row#1#2{...}’, you are allowed to put spaces between the arguments (e.g., ‘\row x n’), because TeX doesn’t use single spaces as undelimited arguments.

In the double dangeorus bend paragraph before exercise 20.5 of the TeXbook you find the sentence:

How does TeX determine where an argument stops, you ask. Answer: [...] An undelimited parameter is followed immediately in the ⟨parameter text⟩ by a parameter token, or it occurs at the very end of the parameter text; in this case the corresponding argument is the next nonblank token, unless that token is ‘{’, when the argument will be the entire {...} group that follows.

In the TeXbook I did find precise definitions neither for the term "single space" nor for the term "nonblank token".

Please enumerate all tokens which TeX does not use as undelimited arguments (unless nested between an explicit character token of category code 1 and an explicit character token of category code 2).

By now I found that TeX does not use explict character tokens of category code 10 and character code 32 as undelimited arguments - you need to focus on what TeX takes for \macro's second argument:

\def\macro#1#2{\def\macrob{Arg 1:(#1) Arg 2:(#2)}}
\macro A B 
\show\macrob
\bye

Implicit character tokens of category code 10 and character code 32 are used as undelimited macro arguments:

\def\macro#1#2{\def\macrob{Arg 1:(#1) Arg 2:(#2)}}
\catcode`\X=13
\uppercase{\let\space= } %
\uppercase{\letX= } %
\macro A\space B 
\show\macrob
\macro AXB
\show\macrob
\bye

Explicit funny spaces are used as undelimited macro arguments:

\def\macro#1#2{\def\macrob{Arg 1:(#1) Arg 2:(#2)}}
\uccode`\ =`\a
\uppercase{\macro A B}%
\show\macrob
\bye

Implicit funny spaces are used as undelimited macro arguments:

\def\macro#1#2{\def\macrob{Arg 1:(#1) Arg 2:(#2)}}
\def\letcs#1#2{\let#1= #2}%
\catcode`\X=13
\uccode`\ =`\a
\uppercase{\letcs\space{ }}%
\uppercase{\letcsX{ }}%
\macro A\space B 
\show\macrob
\macro AXB
\show\macrob
\bye

Implicit/explicit character tokens of category code 12 and character code 32 are used as undelimited macro arguments:

\def\macro#1#2{\def\macrob{Arg 1:(#1) Arg 2:(#2)}}
\catcode`\ =12\relax%
\let\space= %
\macro{A} {B}%
\show\macrob
\macro{A}\space{B}%
\show\macrob
\bye

Control-space is used as undelimited macro argument:

\def\macro#1#2{\def\macrob{Arg 1:(#1) Arg 2:(#2)}}
\macro A\ B
\show\macrob
\bye

So I tested a few cases, but testing edge cases does neither lead to a precise definition for the term "single space" nor lead to a precise definition for the term "nonblank token". ;-)

In other words: I don't know precisely which tokens TeX does not use as undelimited arguments (unless nested between a character token of category code 1 and a character token of category code 2).

It seems the quantity ⟨space token⟩ is not equal to "single space"/"nonblank token":

The TeXbook says in Chapter 24: Summary of Vertical Mode:

The quantity ⟨space token⟩, which was used in the syntax of ⟨optional spaces⟩ above, stands for an explicit or implicit space. In other words, it denotes either a character token of category 10, or a control sequence or active character whose current meaning has been made equal to such a token by \let or \futurelet.

The mentioned "control sequence or active character", subsumeable under ⟨space token⟩, will be used as undelimited macro argument—the examples above show it—while "single space"/"nonblank token" is not used as undelimited macro argument.

Probably "single space"/"nonblank token" is a strict subset of ⟨space token⟩ ?

If so—which subset thereof exactly?

6
  • I think it is just character tokens with catcode 10, but i'd have to chase the tex book or tex-the-program to find a definitive reference. Sep 21, 2020 at 21:40
  • Which character tokens with catcode 10 exactly? My examples show that, e.g., "funny spaces", i.e., character tokens of category code 10 but character code differing from 32 are used as undelimited macro arguments. ("funny spaces" cannot be created by tokenizing .tex-input but as upper- or lowercasing aims at character codes they can be created by upper- or lowercasing a character-token of category code 10 and character code 32.)
    – Jewdokija
    Sep 21, 2020 at 21:45
  • sorry, just normal space, 32 Sep 21, 2020 at 21:45
  • You don't need to say sorry. :-) But I wish to say "thank you" to you. So: Thank you. :-)
    – Jewdokija
    Sep 21, 2020 at 21:47
  • @DavidCarlisle By the way: How does one chase tex-the-program? Do you mean looking into Donald Ervin Knuth's book "Computers & Typesetting, Volume B: TeX: The Program" ?
    – Jewdokija
    Sep 21, 2020 at 21:50

3 Answers 3

8

The style of the TeXbook is often to say something which is correct, but not the complete truth.

There is no formal definition of a “single space”, because it's not needed.

Indeed, if you try

\begingroup\def\\{\global\let\spacetoken= }\\ \endgroup

\def\foo#1#2{(First is #1)(Second is #2)}

\foo AB

\foo A B

\edef\two{\space\space}
\expandafter\foo\expandafter A\two B

\foo A\spacetoken B

\bye

you'll get three instances of

(First is A)(Second is B)

and the last line will instead produce

(First is A)(Second is )B

The \expandafter trick is used to inject multiple spaces between A and B. So you see that the next exercise is “more correct”: TeX skips any explicit space token when looking for an undelimited argument.

The last example shows that implicit space tokens are not skipped. The first line in the code is borrowed from exercise 24.6, to make \spacetoken an implicit space token, because one cannot simply do like in \let\bgroup={. If you add \show\spacetoken you get

> \spacetoken=blank space  .

but this is not ignored when looking for an undelimited argument.

An explicit space token is a character token of category code 10 (space or tab, under normal setting; but see later for more details if you're interested in them). Under normal setting this might be generated by a blank space or a tab in the input, or any character which is assigned category code 10 at the time when input is tokenized.


But, there is a catch. There's always one!

One has to take into account that TeX will absorb characters with category code 10 assigning them character code 32 independently on their original character code. Thus tabs are not different from spaces, because they are the same once tokenization has been performed.

So, what's the problem with

\uccode` =`x \uppercase{\foo A B}

that doesn't ignore the funny space? It's in fact different from

\catcode`*=10 \foo A*B

that ignores the asterisk, because it has category code 10.

The fact is that characters with category code 10 are normalized to have character code 32 during tokenization. However, when \uppercase is applied, tokenization has already been performed and the space has character code 32. But after \uppercase the character becomes x10, which no longer is valid for being ignored, because it doesn't have character code 32.

Hence the answer that only character with character code 32 and category code 10 are ignored is correct, but misleading if the normalization is not taken into account.

6
  • 1
    I believe \expandafter\foo\expandafter A\two B would be sufficient after the \edef line. Trying to expand non-active character token A? :-)
    – frougon
    Sep 22, 2020 at 9:01
  • 1
    @frougon You're completely right
    – egreg
    Sep 22, 2020 at 9:06
  • @UlrichDiez Right, and I added a final full description based on the example. The fact that pairs (32,10) are ignored must take into account the normalization.
    – egreg
    Sep 22, 2020 at 13:07
  • Now your answer says: An explicit space token is a character token of category code 10. ... TeX skips any explicit space token when looking for an undelimited argument. But this is not correct: TeX only skips explicit space tokens of character code 32 when looking for an undelimited argument. When you do \uccode`\ =`\a\uppercase{\macro{A} {B}}\show\macrob, then you will see that an a is used as \macro's second argument. That a is an explicit space-token: It is an explicit character token. It has category code 10. But it does not have character code 32. ;-) Sep 22, 2020 at 13:18
  • @UlrichDiez Funny spaces are a very advanced concept. The answer is divided into two parts, initially for “normal” input.
    – egreg
    Sep 22, 2020 at 13:44
5

tex.web has

begin if cur_tok=space_token then

to skip over the ignored tokens where space_token is

@d space_token=@'5040 {$2^8\cdot|spacer|+|" "|$}
6
  • What does "@d space_token=@'5040 {$2^8\cdot|spacer|+|" "|$}" mean???????
    – Jewdokija
    Sep 21, 2020 at 22:22
  • @Jewdokija my paraphrasing it the first time was wrong so I deleted:-) it's getting late here, '5040 is octal so hex A20 which is basically a normal space catcode hex A =10 and charcode hex 20 = 32 (the bit in braces is the tex source code for the comment for this line of text) Sep 21, 2020 at 22:28
  • 2
    I still don't know what this is all about. After looking at it for a while I assume that this has something to do with a formula for representing a catcode/charcode-pair: Seems space_token is a constant whose binary representation can be splitted in two bytes, the more significant byte holding the binary representation of "spacer", the less significant byte holding the character-code/ASCII-code of the space-character. Solving 5040_(8)=2^8 · spacer + 32 for spacer: spacer = 10. 10 is the category code "space". So the formula for representing character-tokens is (2^8)(catcode)+(charcode). Right?
    – Jewdokija
    Sep 21, 2020 at 22:54
  • 2
    So as section 393.<Store the current token, but goto continue if it is a blank space that would become an undelimited parameter 393 > ≡ (begin if cur_tok=space_token then... only checks for space_token this tells me that the token behind space_token, that is the catcode10/charcode32-pair, is the only token which is not stored for an undelimited argument?
    – Jewdokija
    Sep 21, 2020 at 23:07
  • 2
    TeXbook, right after exercise 20.2, says: "Control sequences can be defined in terms of parameters, and you can supply arguments that will be substituted for the parameters." So why in TeX - The Program , section 393, the term "parameter" is used instead of "argument"? That section is not about defining control sequences but is about processing things/arguments supplied by the user.
    – Jewdokija
    Sep 21, 2020 at 23:24
0

Sequences of explicit character tokens of character code 32 and category 10(space) are the only things that TeX skips while "looking" for the begin of an undelimited argument.


The quantity ⟨space token⟩ indeed is not equal to "single space"/"nonblank token" in the sense of the TeXbook-paragraphs quoted by you:

The quantity ⟨one optional space⟩ is defined as:

⟨one optional space⟩⟨space token⟩ | ⟨empty⟩

Wherever ⟨one optional space⟩ is allowed, that can as well be an implicit space token.

See, e.g.,

\lowercase{\let\sptoken = } %
\edef\result{\number1234 }
\show\result
\edef\result{\number1234\sptoken}
\show\result
\let\result\sptoken\sptoken=\sptoken\TeX
\bye

(Here \lowercase does nothing but remove the braces. This way you get two explicit space tokens of character code 32 behind "=". The first one will be discarded because with \let-assignments one space is optional behind "=". The second one will not be discarded but will be the token whose meaning is assigned to \sptoken.)

\sptoken is an implicit space token.
It is discarded during TeX's \number-evaluation like an explicit space token.
It is also discarded like any other ⟨optional space⟩ while performing the second \let-assignment.
But TeX would not skip \sptoken while "looking" for the beginning of an undelimited argument.

So this example proves that the quantity ⟨space token⟩ is not equal to "single space"/"nonblank token" in the sense of the TeXbook-paragraphs quoted by you.


By the way:

Your question is focused on how TeX treats tokens while looking for the begin of an undelimited argument.

Your question refers to a stage of processing where tokenization is already done.

Nevertheless it is worth to mention a fact related to the process of tokenizing .tex-input:

If, during the process of tokenizing .tex-input, TeX encounters a character whose category code is 10 (space) while the reading-apparatus is in state M(middle of line), then TeX will append an explicit character token of category 10(space) and character code 32 to the token-stream. I.e., the resulting token will have character code 32 regardless the number which the code point of the character of the input in question has.

E.g., the horizontal tab—horizontal tab has code-point number 9 in ASCII—usually has category code 10 assigned, too. Therefore tokenizing a horizontal tab usually yields an explicit character token of category 10(space) and character code 32. I.e., that very token which is skipped by TeX while "looking" for the begin of an undelimited argument.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .