When an implicit token is allowed?

Question

Example: the latex code:

\documentclass{article}
\begin{document}

% test-case 1
% Output: "b"
% Expansion is not allowed, and implicit token doesn't take effect.
\begingroup
\let\implicitToken=b
\uppercase{\implicitToken}
\endgroup

% test-case 2
% Output: "b"
% Expansion is allowed, and implicit token takes effect.
\begingroup
\let\implicitToken=b
\if \implicitToken b
b
\else
B
\fi
\endgroup

% test-case 3
% Compile error: \cq is undefined
% Expansion is not allowed, and implicit token doesn't take effect.
\begingroup
\let\implicitToken\cq
\def\implicitToken{}
\cq
\endgroup

% test-case 4
% Output: "B"
% Expansion is allowed, but implicit token doesn't take effect.
\begingroup
\let\implicitToken=b
\edef\cq{\implicitToken}
\let\implicitToken=B
\cq
\endgroup

\end{document}

Here are my questions:

• Whether the implicit token is allowed if expansion is allowed in the context?

  This might be the only way to explain the test-case 1, 2, 3.But it can not explain test-case 4.

• If not, when it is allowed?

---

I have read the TeXByTopic but I can't find a certain conclusion.

Answer 1

Implicit tokens are not expanded to their meaning, they are their meaning. So with

\let\tokenAlias=b

\tokenAlias is an b. That shows up in your second test: \tokenAlias and b are entirely equivalent.

However, case changing takes place based on the character code of the token concerned. (As noted in a comment, \uppercase is also not expandable.) Thus \tokenAlias is not affected by \uppercase as it does not have a character code at all, being a control sequence.

The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.

---

Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:

1. \implicitToken remains unchanged as it does not have a character code in the upper case table

2. \implicitToken is b, so comparing the two using \if is true: there is no expansion of \implicitToken.

3. \cq is never defined, and so the \let and \def lines have no effect

4. The \edef does nothing as \implicitToken is not expanded. Rather, we end up with \cq with replacement text \implicitToken: the effect this has thus depends on the meaning at point-of-use.

Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.

Answer 2

Let's analyze your tests one by one.

An implicit token (more precisely an implicit character token) is a symbolic token defined by \let<token>=<character>.

Test 2

% test-case 2
% Output: "b"
% Expansion is enabled, and implicit token takes effect.
\begingroup
\let\implicitToken=b
\if \implicitToken b
b
\else
B
\fi
\endgroup

This outputs “b”, because an implicit token behaves like the character it is defined to be equal to in the context of \if or \ifcat.

Test 3

% test-case 3
% Compile error: \cq is undefined
% Expansion is disabled, and implicit token doesn't take effect.
\begingroup
\let\implicitToken\cq
\def\implicitToken{}
\cq
\endgroup

This is easier: the assignment \def\implicitToken{} overrides the \let. On the other hand, \implicitToken is never used and \cq (undefined) raises an error.

Test 4

% test-case 4
% Output: "B"
% Expansion is enabled, but implicit token doesn't take effect.
\begingroup
\let\implicitToken=b
\edef\cq{\implicitToken}
\let\implicitToken=B
\cq
\endgroup

If you add \show\cq after doing the \edef you'll see

> \cq=macro:
->\implicitToken .

and the output will contain a “B”, because of the subsequent redefinition.

An implicit token is not expandable, so the codes

\let\implicitToken=b
\edef\cq{\implicitToken}

and

\let\implicitToken=b
\def\cq{\implicitToken}

are completely equivalent. TeX will use the meaning of \implicitToken current at expansion of \cq time: in the case of your test it is “B”.

Test 1

% test-case 1
% Output: "b"
% Expansion is disabled, and implicit token doesn't take effect.
\begingroup
\let\implicitToken=b
\uppercase{\implicitToken}
\endgroup

The primitive \uppercase examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the \uccode vector). It doesn't act on implicit tokens.

Comments

There is an asymmetry: with \let\implicitToken=b, the test \if\implicitToken b will return true. Similarly \ifcat will use the category code of the character current at definition time; for instance

\let\implicitToken=b
\catcode`b=12
\ifcat\implicitToken a%
  \message{letter}%
\else
  \message{nonletter}%
\fi

will print letter on the console, whereas \ifcat ba would print nonletter.

Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with \bgroup and \egroup, which can be used as substitutes for { and } in certain (not all) contexts. I guess that making them subject to transformations in \uppercase or \lowercase would be irksome.

Expansion

Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive \dimen is not expandable. If TeX finds \dimen2=12pt, it will pass \dimen to the execution processor; the execution prompts the search in the input stream for a <number>, an <optional equal> and a suitable value (with expansion). On the contrary, in \advance\dimen2 by 2pt, the token \dimen is absorbed as the argument to \advance for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.

What tokens are expandable? Easy:

1. some primitives, such as conditionals, \string, \noexpand and others;
2. macros, that is, tokens defined with \def, \edef, \gdef or \xdef;
3. tokens \let equal to an expandable token.

Since your \implicitToken belongs to none of the above types, it is unexpandable.