Which tokens does TeX not use as undelimited arguments (unless between { and } )?

Question

Which tokens does TeX not use as undelimited arguments (unless nested between an explicit character token of category code 1 and an explicit character token of category code 2)?

In the last but one dangerous bend paragraph before exercise 20.4 of the TeXbook you find the sentence:

After you have said ‘\def\row#1#2{...}’, you are allowed to put spaces between the arguments (e.g., ‘\row x n’), because TeX doesn’t use single spaces as undelimited arguments.

In the double dangeorus bend paragraph before exercise 20.5 of the TeXbook you find the sentence:

How does TeX determine where an argument stops, you ask. Answer: [...] An undelimited parameter is followed immediately in the ⟨parameter text⟩ by a parameter token, or it occurs at the very end of the parameter text; in this case the corresponding argument is the next nonblank token, unless that token is ‘{’, when the argument will be the entire {...} group that follows.

In the TeXbook I did find precise definitions neither for the term "single space" nor for the term "nonblank token".

Please enumerate all tokens which TeX does not use as undelimited arguments (unless nested between an explicit character token of category code 1 and an explicit character token of category code 2).

By now I found that TeX does not use explict character tokens of category code 10 and character code 32 as undelimited arguments - you need to focus on what TeX takes for \macro's second argument:

\def\macro#1#2{\def\macrob{Arg 1:(#1) Arg 2:(#2)}}
\macro A B 
\show\macrob
\bye

Implicit character tokens of category code 10 and character code 32 are used as undelimited macro arguments:

\def\macro#1#2{\def\macrob{Arg 1:(#1) Arg 2:(#2)}}
\catcode`\X=13
\uppercase{\let\space= } %
\uppercase{\letX= } %
\macro A\space B 
\show\macrob
\macro AXB
\show\macrob
\bye

Explicit funny spaces are used as undelimited macro arguments:

\def\macro#1#2{\def\macrob{Arg 1:(#1) Arg 2:(#2)}}
\uccode`\ =`\a
\uppercase{\macro A B}%
\show\macrob
\bye

Implicit funny spaces are used as undelimited macro arguments:

\def\macro#1#2{\def\macrob{Arg 1:(#1) Arg 2:(#2)}}
\def\letcs#1#2{\let#1= #2}%
\catcode`\X=13
\uccode`\ =`\a
\uppercase{\letcs\space{ }}%
\uppercase{\letcsX{ }}%
\macro A\space B 
\show\macrob
\macro AXB
\show\macrob
\bye

Implicit/explicit character tokens of category code 12 and character code 32 are used as undelimited macro arguments:

\def\macro#1#2{\def\macrob{Arg 1:(#1) Arg 2:(#2)}}
\catcode`\ =12\relax%
\let\space= %
\macro{A} {B}%
\show\macrob
\macro{A}\space{B}%
\show\macrob
\bye

Control-space is used as undelimited macro argument:

\def\macro#1#2{\def\macrob{Arg 1:(#1) Arg 2:(#2)}}
\macro A\ B
\show\macrob
\bye

So I tested a few cases, but testing edge cases does neither lead to a precise definition for the term "single space" nor lead to a precise definition for the term "nonblank token". ;-)

In other words: I don't know precisely which tokens TeX does not use as undelimited arguments (unless nested between a character token of category code 1 and a character token of category code 2).

It seems the quantity ⟨space token⟩ is not equal to "single space"/"nonblank token":

The TeXbook says in Chapter 24: Summary of Vertical Mode:

The quantity ⟨space token⟩, which was used in the syntax of ⟨optional spaces⟩ above, stands for an explicit or implicit space. In other words, it denotes either a character token of category 10, or a control sequence or active character whose current meaning has been made equal to such a token by \let or \futurelet.

The mentioned "control sequence or active character", subsumeable under ⟨space token⟩, will be used as undelimited macro argument—the examples above show it—while "single space"/"nonblank token" is not used as undelimited macro argument.

Probably "single space"/"nonblank token" is a strict subset of ⟨space token⟩ ?

If so—which subset thereof exactly?

Answer 1

The style of the TeXbook is often to say something which is correct, but not the complete truth.

There is no formal definition of a “single space”, because it's not needed.

Indeed, if you try

\begingroup\def\\{\global\let\spacetoken= }\\ \endgroup

\def\foo#1#2{(First is #1)(Second is #2)}

\foo AB

\foo A B

\edef\two{\space\space}
\expandafter\foo\expandafter A\two B

\foo A\spacetoken B

\bye

you'll get three instances of

(First is A)(Second is B)

and the last line will instead produce

(First is A)(Second is )B

The \expandafter trick is used to inject multiple spaces between A and B. So you see that the next exercise is “more correct”: TeX skips any explicit space token when looking for an undelimited argument.

The last example shows that implicit space tokens are not skipped. The first line in the code is borrowed from exercise 24.6, to make \spacetoken an implicit space token, because one cannot simply do like in \let\bgroup={. If you add \show\spacetoken you get

> \spacetoken=blank space  .

but this is not ignored when looking for an undelimited argument.

An explicit space token is a character token of category code 10 (space or tab, under normal setting; but see later for more details if you're interested in them). Under normal setting this might be generated by a blank space or a tab in the input, or any character which is assigned category code 10 at the time when input is tokenized.

---

But, there is a catch. There's always one!

One has to take into account that TeX will absorb characters with category code 10 assigning them character code 32 independently on their original character code. Thus tabs are not different from spaces, because they are the same once tokenization has been performed.

So, what's the problem with

\uccode` =`x \uppercase{\foo A B}

that doesn't ignore the funny space? It's in fact different from

\catcode`*=10 \foo A*B

that ignores the asterisk, because it has category code 10.

The fact is that characters with category code 10 are normalized to have character code 32 during tokenization. However, when \uppercase is applied, tokenization has already been performed and the space has character code 32. But after \uppercase the character becomes x₁₀, which no longer is valid for being ignored, because it doesn't have character code 32.

Hence the answer that only character with character code 32 and category code 10 are ignored is correct, but misleading if the normalization is not taken into account.

Answer 2

tex.web has

begin if cur_tok=space_token then

to skip over the ignored tokens where space_token is

@d space_token=@'5040 {$2^8\cdot|spacer|+|" "|$}

Answer 3

Sequences of explicit character tokens of character code 32 and category 10(space) are the only things that TeX skips while "looking" for the begin of an undelimited argument.

---

The quantity ⟨space token⟩ indeed is not equal to "single space"/"nonblank token" in the sense of the TeXbook-paragraphs quoted by you:

The quantity ⟨one optional space⟩ is defined as:

⟨one optional space⟩ → ⟨space token⟩ | ⟨empty⟩

Wherever ⟨one optional space⟩ is allowed, that can as well be an implicit space token.

See, e.g.,

\lowercase{\let\sptoken = } %
\edef\result{\number1234 }
\show\result
\edef\result{\number1234\sptoken}
\show\result
\let\result\sptoken\sptoken=\sptoken\TeX
\bye

(Here \lowercase does nothing but remove the braces. This way you get two explicit space tokens of character code 32 behind "=". The first one will be discarded because with \let-assignments one space is optional behind "=". The second one will not be discarded but will be the token whose meaning is assigned to \sptoken.)

\sptoken is an implicit space token.
It is discarded during TeX's \number-evaluation like an explicit space token.
It is also discarded like any other ⟨optional space⟩ while performing the second \let-assignment.
But TeX would not skip \sptoken while "looking" for the beginning of an undelimited argument.

So this example proves that the quantity ⟨space token⟩ is not equal to "single space"/"nonblank token" in the sense of the TeXbook-paragraphs quoted by you.

---

By the way:

Your question is focused on how TeX treats tokens while looking for the begin of an undelimited argument.

Your question refers to a stage of processing where tokenization is already done.

Nevertheless it is worth to mention a fact related to the process of tokenizing .tex-input:

If, during the process of tokenizing .tex-input, TeX encounters a character whose category code is 10 (space) while the reading-apparatus is in state M(middle of line), then TeX will append an explicit character token of category 10(space) and character code 32 to the token-stream. I.e., the resulting token will have character code 32 regardless the number which the code point of the character of the input in question has.

E.g., the horizontal tab—horizontal tab has code-point number 9 in ASCII—usually has category code 10 assigned, too. Therefore tokenizing a horizontal tab usually yields an explicit character token of category 10(space) and character code 32. I.e., that very token which is skipped by TeX while "looking" for the begin of an undelimited argument.