Syntax of TeX for primitives that demand braces

Question

TeX defines some primitives which some of them expand (or try) the following tokens before execute itself; others even require argument between braces. But I couldn't get the point in this case.

\uppercase\expandafter{\romannumeral\year} % a cut off from https://www.tug.org/utilities/plain/cseq.html#uppercase-rp

\uppercase demands an argument in braces (run \uppercase abcdef and you will understand me). But in the highlighted example above, \uppercase is getting a expandable(?) primitive as argument and it's correctly executed.

Trying to looking more closely I notice that

\def\seq{abcdef}%
\uppercase\seq  %

generates a error, but

\def\seq{{abcdef}}%
\uppercase\seq    %

doesn't.

I am missing a point. Which one? Could some explain such behaviors? Are braces considered as special or a restricted case of expandables?

Answer 1

That's an interesting part of TeX. Some primitives require an argument in braces. Some examples are \uppercase, \vbox, \halign.

However, they work in a different way than macros. For instance, the open brace can also be implicit (\bgroup is an implicit open brace) and they do full expansion of the following tokens until finding the open brace. In the process, \relax and space tokens are discarded.

To be more precise, \vbox, \hbox and \halign also accept keywords and values before the open brace. So, if you do \def\test#1{to #1pt}, you can do

\vbox\test{90}{<balanced text>}

and this will result in \vbox to 90pt{<balanced text>}. Beware that some of those primitives accept an implicit closed brace, but some don't. In the first category there are \vbox, \hbox and \halign, because the <material> is processed until finding the matching closed brace; in the second category you find \uppercase, \lowercase and \toks, because they do no processing at all to the <balanced text>. Besides \toks you also find all primitive token registers such as \everypar and all \toksdef tokens.

This is the mechanism that allows for

\uppercase\expandafter{\romannumeral 42}

that results in XXXXII. Indeed, \uppercase triggers expansion of the following token and the expansion of \expandafter is vacuous after having triggered a one step expansion of the second token following it (with no macro expansion taking place when looking for the two tokens).

Thus with

\def\testA{abc}
\def\testB{{abc}}
\uppercase\testA
\uppercase\testB

the first call will fail because after macro expansion it results in \uppercase abc, but the second one succeeds because it results in \uppercase{abc} which is legal. Note that the closing brace is not implicit, because it is put in the main stream explicitly when expanding \testB.

Why are \relax and space tokens ignored? Suppose you do

\def\myspace{50pt}

that you want to use in the context of \hskip\myspace, you might incur in problems with something like

We have \hskip\myspace\space minuscule chances

(try it). So a better definition would be

\def\mysspace{50pt\relax}

and the example above would go through. But you might also want to do

\hbox to \myspace{<text>}

and the \relax would not be good, weren't it for the rule that when looking for { such token is ignored. It is called a <filler> in the TeXbook.

So, no, braces are not “kind of expandable”.

Answer 2

Your experiments with \uppercase\seq were the big clue here. \uppercase wants its argument to be enclosed in braces so if it finds a control sequence after it, it will expand it in search of its opening brace. So, with \uppercase\seq, it expands \seq and so

\def\seq{abcdef}%
\uppercase\seq  %

is the same as

\uppercase abcdef

while

\def\seq{{abcdef}}%
\uppercase\seq    %

is equivalent to

\uppercase{abcdef}

On the other hand, when you have

\uppercase\expandafter{\romannumeral\year} % a cut off from https://www.tug.org/utilities/plain/cseq.html#uppercase-rp

the \expandafter gets expanded, which means it looks past the curly brace, expands \romannumeral\year (which will give ccxxi for the next 81 days) and \uppercase will then operate on

\uppercase{ccxxi}

Without the \expandafter, however,

\uppercase{\romannumeral\year}

will see nothing it can uppercase and the output will not be CCXXI but ccxxi.

Answer 3

Let's look at TeXbook, Chapter 24: Summary of Vertical Mode—there you find the following snippets of information:

\uppercase⟨general text⟩, \lowercase⟨general text⟩. The ⟨balanced text⟩ in the ⟨general text⟩ is converted to uppercase form or to lowercase form using the \uccode or \lccode table, as explained in Chapter 7; no expansion is done. Then TeX will read that ⟨balanced text⟩ again.

(For the sake of precision instead of just "; no expansion is done" it should be "; with the ⟨balanced text⟩ no expansion is done".)

⟨general text⟩→⟨filler⟩{⟨balanced text⟩⟨right brace⟩; the left brace token can be implicit.
⟨filler⟩→⟨optional spaces⟩  |  ⟨filler⟩\relax⟨optional spaces⟩

⟨optional spaces⟩→⟨empty⟩  |  ⟨space token⟩⟨optional spaces⟩
⟨empty⟩→nothing
⟨space token⟩→"either a character token of category 10, or a control sequence or active character whose current meaning has been made equal to such a token by \let or \futurelet."

⟨balanced text⟩ is a token list "where ⟨left brace⟩ and ⟨right brace⟩ tokens are properly nested like parentheses" if present.

"⟨left brace⟩ and ⟨right brace⟩ are explicit character tokens whose category codes are respectively of types 1 and 2". (1=Beginning of group. 2=End of group.)

Let's look at TeXbook, Chapter 20: Definitions (also called Macros)—there you find the following snippets of information:

Here now is the promised list of all cases when expandable tokens are not expanded.
[...]

• When TeX is absorbing the replacement text of a \def or \gdef or \read; or the text of a token variable like \everypar or \toks0; or the token list for \uppercase or \lowercase or \write. (The token list for \write will be expanded later, when it is actually output to a file.)

[...]

(Note that in the list of cases where expandable tokens are not expanded/where expansion is suppressed the case of TeX scanning for the left brace token of a ⟨general text⟩ is not an item on its own.)

Let's be very picky when looking at the "picture" formed by these snippets of information:

In the list of cases where expandable tokens are not expanded/where expansion is suppressed the case of TeX scanning for the left brace token of a ⟨general text⟩ is not an item on its own.

Thus with \uppercase/\lowercase expansion is not suppressed while scanning for the left brace token belonging to the following ⟨general text⟩: TeX keeps expanding expandable tokens until either for some reason an error-message needs to be raised or TeX has found the ⟨general text⟩'s left brace token whereafter ⟨balanced text⟩ follows, in turn trailed by a ⟨right brace⟩. That ⟨balanced text⟩ is a token list with the property of ⟨left brace⟩ and ⟨right brace⟩ tokens being properly nested like parentheses if present. That's why in chapter 20 that ⟨balanced text⟩ is referred to by means of the term "token list".
According to chapter 20 the tokens forming that ⟨balanced text⟩ will not be expanded further while \uppercase/\lowercase does its work.

---

The need to be extremely picky about every phrase in the TeXbook when it comes to making an overall picture from individual pieces of information that may occur in widely separated sections of the book sometimes leads me to suggest that one read the TeXbook like a good lawyer would review a contract, looking closely and picky at aspects and correlations presented to discover and avoid potential pitfalls in the form of possible interpretive errors.

Knuth explains in the preface of the TeXbook that not always all information belonging to a context is given immediately, but little by little:

Another noteworthy characteristic of this manual is that it doesn’t always tell the truth. When certain concepts of TeX are introduced informally, general rules will be stated; afterwards you will find that the rules aren’t strictly true. In general, the later chapters contain more reliable information than the earlier ones do. The author feels that this technique of deliberate lying will actually make it easier for you to learn the ideas. Once you understand a simple but false rule, it will not be hard to supplement that rule with its exceptions.

---

With

\uppercase\expandafter{\romannumeral\year}%

TeX scans for \uppercase's ⟨general text⟩'s left brace token whereby expansion is not suppressed. Thus \expandafter is expanded. The result of expanding \expandafter is the result of expanding \romannumeral. \romannumeral in turn triggers scanning for a TeX-⟨number⟩-quantity whereby \year is found and expanded.

So in the end as the ⟨general text⟩'s ⟨balanced text⟩, which comes behind the ⟨general text⟩'s left brace token({) and before the ⟨general text⟩'s ⟨right brace⟩(}), you get explicit character-tokens of category-code 12(other) representing the number of the year in lowercase roman notation.
According to chapter 20 there are no attempts at expanding that ⟨balanced text⟩ any further while \uppercase "works" on it. (In this case such attempts would be futile as in this case the ⟨balanced text⟩ consists only of non-expandable explicit character-tokens.)

---

With

\def\seq{abcdef}%
\uppercase\seq

TeX scans for \uppercase's ⟨general text⟩'s left brace token. Hereby expansion is not suppressed. Thus \seq is expanded. Now TeX does find neither another expandable token nor ⟨filler⟩ nor a left brace token belonging to ⟨general text⟩ but does find an explicit character-token a of category code 11(letter) and therefore raises an error.

---

With

\def\seq{{abcdef}}%
\uppercase\seq

TeX scans for \uppercase's ⟨general text⟩'s left brace token. Hereby expansion is not suppressed. Thus \seq is expanded. Thus TeX does find the left brace token of the sequence {abcdef} and takes it for the left brace token of ⟨general text⟩. abcdef of that sequence is taken for the ⟨balanced text⟩ of the ⟨general text⟩. } of that sequence is taken for the ⟨right brace⟩ of the ⟨general text⟩. So the syntax is okay and no error needs to be raised.

---

Due to the ⟨filler⟩-thing with ⟨general text⟩ you can also do

\def\seq{ \relax {abcdef}}%
\uppercase\seq    %

While scanning for \uppercase's ⟨general text⟩'s left brace token both the explicit space token which comes into being right behind the left brace of \seq's definition and the subsequent \relax will be discarded as ⟨filler⟩.

---

Be aware that, besides \uppercase and \lowercase, there are many more primitives in TeX which process ⟨general text⟩ whereby, before attempting to do the actual work, expansion is not suppressed until finding the ⟨general text⟩'s left brace token.