What are the constituents of a LaTeX3 token list?

Question

I'm trying to understand what a LaTeX3 token list is. Unfortunately, the LaTeX3 documentation is anything but helpful. According to the LaTeX3 Interfaces documentation token lists can be viewed in two ways (p. 35):

A token list (explicit, or stored in a variable) can be seen either as a list of “items”, or a list of “tokens”. An item is whatever \use:n would grab as its argument: a single non-space token or a brace group, with optional leading explicit space characters (each item is thus itself a token list). A token is either a normal N argument, or ␣, {, or } (assuming normal TeX category codes).

What is a LaTeX3 token?

According to the above excerpt

A token is either a normal N argument, or ␣, {, or } (assuming normal TeX category codes).

1. What is "a normal N argument"?
2. Consider the phrase "␣, {, or } (assuming normal TeX category codes)". Is it the same as saying "the TeX tokens (character code = 32, category code = 10), (character code = 123, category code = 1), or (character code = 125, category code = 2)"? Or is it the same as saying "any TeX token, whose category code is one of 1, 2, or 10"?

What is a LaTeX3 item?

According to the above excerpt

An item is whatever \use:n would grab as its argument: a single non-space token or a brace group, with optional leading explicit space characters (each item is thus itself a token list).

1. What does "a single non-space token" mean? does the word "token" refer to a TeX token, or to a LaTeX3 token? Does the expression "non-space" refer to any TeX token that is not (character code = 32, category code = 10), or to any TeX token whose category code is not 10, or to any TeX token whose character code is not 32? What about other whitespace characters, such as a newline? Do they count?

2. What is a "brace group"? Does it include the delimiting braces? Does it include the whitespace contained inside?

3. Does the phrase "with optional leading explicit space characters" refer only to the brace group, or also to the single non-space token? What does it mean explicit space characters? Are the leading explicit space character part of the item? What about non-explicit space characters?

Answer 1

In expl3 lingo, token is the same as in basic TeX, see What is a token?

However, a token list (in a variable or explicit) can be seen in a dual fashion:

1. a container of tokens
2. a list of items

For instance, if you do \tl_use:N \l_tmpa_tl or pass a token list as a V argument (for a variable) or as an n argument (usually braced), the mode of operation is 1; when you pass it to \tl_map_inline:Nn or \tl_map_function:NN (for variables) or \tl_map_inline:nn or \tl_map_function:nN (for explicit token lists, usually braced), mode 2 is used.

What is the difference?

An explicit space token is a character token of category 10 (any character code). In expl3 code (after \ExplSyntaxOn), the space character has category code 9 (ignored), but ~ has category code 10, so it counts as an explicit space token.

Now, suppose you do

\tl_set:Nn \l_tmpa_tl {~{ab}~c~\foo}

the token list variable will consist of nine tokens and three items. Thus a call of \tl_map_inline:Nn \l_tmpa_tl {<code>} will perform three steps, because explicit space tokens will be ignored because of TeX parsing rules, which make TeX ignore explicit space tokens when looking for an undelimited macro argument.

The token \c_space_token is of course not ignored in this context, because it is an implicit space token (the same as \@sptoken in traditional LaTeX), nor \c_space_tl would be, because it is a token list variable containing an explicit space token.

The distinction about N and n arguments is a LaTeX3 convention. When a function is defined with an N argument type, the programmer should always pass it a single token, when it has an n argument type the programmer should pass a (balanced) braced group {<tokens>}. However, it's impossible to fully enforce the convention, because we're reading TeX code, after all.

So if we have the function \foo_bar:Nn, a correct call should be like

\foo_bar:Nn \cs {abc}

where \cs stands for a control sequence (but it could also be a single character token). Because of TeX parsing rules, the call \foo_bar:Nn {\cs}{abc} would be equivalent, but it would be against LaTeX3 good practice.

Your questions.

1. A normal N argument is a single token, but explicit space tokens would be ignored; if you call \foo_bar:Nn ~ \cs {abc}, the first argument will still be \cs.

2. A brace group is what you pass as an undelimited argument consisting of more tokens, as described in the TeXbook; the braces will be stripped off when the argument replaces the corresponding parameter token in the replacement text of the macro

3. An item in a token list can be preceded by (explicit) space tokens, which will be ignored when the token list is used in mode 2.

---

One of your comments above mentions

{ Hello } ~ world

If this is encountered when \ExplSyntaxOn is in force, this consists of 13 tokens, precisely

{₁ H₁₁ e₁₁ l₁₁ l₁₁ o₁₁ }₂ ~₁₀ w₁₁ o₁₁ r₁₁ l₁₁ d₁₁

because the space character has category code 9, so it's ignored during tokenization. That list has six items

{Hello} w o r l d

Tabs (^^I) are similarly ignored and \endlinechar is set to –1, so also new lines are irrelevant.