# When an implicit token is allowed?

Example: the latex code:

\documentclass{article}
\begin{document}

% test-case 1
% Output: "b"
% Expansion is not allowed, and implicit token doesn't take effect.
\begingroup
\let\implicitToken=b
\uppercase{\implicitToken}
\endgroup

% test-case 2
% Output: "b"
% Expansion is allowed, and implicit token takes effect.
\begingroup
\let\implicitToken=b
\if \implicitToken b
b
\else
B
\fi
\endgroup

% test-case 3
% Compile error: \cq is undefined
% Expansion is not allowed, and implicit token doesn't take effect.
\begingroup
\let\implicitToken\cq
\def\implicitToken{}
\cq
\endgroup

% test-case 4
% Output: "B"
% Expansion is allowed, but implicit token doesn't take effect.
\begingroup
\let\implicitToken=b
\edef\cq{\implicitToken}
\let\implicitToken=B
\cq
\endgroup

\end{document}


Here are my questions:

• Whether the implicit token is allowed if expansion is allowed in the context?

This might be the only way to explain the test-case 1, 2, 3.But it can not explain test-case 4.

• If not, when it is allowed?

I have read the TeXByTopic but I can't find a certain conclusion.

• Welcome to TeX.SX! Yes, that's the interesting thing about expansion and \uppercase is not expandable in itself. – TeXnician Sep 6 '18 at 6:04
• in test-case 4, you let \implicitToken to b which is not-expandable, so \implicitToken does not expand in an \edef. Then you redefine its meaning so the new meaning is used. – user4686 Sep 6 '18 at 8:57
• I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example is \let\bgroup={ \let\egroup=}. – Joseph Wright Sep 6 '18 at 8:58
• you can achieve same as test-case 4 with \let\implicitToken\ignorespaces then do the \edef then do \let\implicitToken\hrule and execute \cq to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for the \hrule to show...) – user4686 Sep 6 '18 at 8:58
• there is no expansion in any of your examples except one step \cq to \implicitToken – David Carlisle Sep 6 '18 at 9:06

Implicit tokens are not expanded to their meaning, they are their meaning. So with

\let\tokenAlias=b


\tokenAlias is an b. That shows up in your second test: \tokenAlias and b are entirely equivalent.

However, case changing takes place based on the character code of the token concerned. (As noted in a comment, \uppercase is also not expandable.) Thus \tokenAlias is not affected by \uppercase as it does not have a character code at all, being a control sequence.

The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.

Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:

1. \implicitToken remains unchanged as it does not have a character code in the upper case table

2. \implicitToken is b, so comparing the two using \if is true: there is no expansion of \implicitToken.

3. \cq is never defined, and so the \let and \def lines have no effect

4. The \edef does nothing as \implicitToken is not expanded. Rather, we end up with \cq with replacement text \implicitToken: the effect this has thus depends on the meaning at point-of-use.

Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.

• \uppercase is also not expandable. So \tokenAlias->l is an expansion? – Leslie Leigh Sep 6 '18 at 7:36
• @LeslieLeigh \tokenAlias is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, where \tokenAlias is l, which then produces the output l. That's not expansion. – Joseph Wright Sep 6 '18 at 7:41
• Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it? – Leslie Leigh Sep 6 '18 at 9:03
• @LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for example \uppercase{b} is the token B but there is no token other than \implicitToken that \uppercase{\implicitToken} could be defined to be. – David Carlisle Sep 6 '18 at 9:09
• @LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic for \uppercase for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise). – Joseph Wright Sep 6 '18 at 11:46

Let's analyze your tests one by one.

An implicit token (more precisely an implicit character token) is a symbolic token defined by \let<token>=<character>.

## Test 2

% test-case 2
% Output: "b"
% Expansion is enabled, and implicit token takes effect.
\begingroup
\let\implicitToken=b
\if \implicitToken b
b
\else
B
\fi
\endgroup


This outputs “b”, because an implicit token behaves like the character it is defined to be equal to in the context of \if or \ifcat.

## Test 3

% test-case 3
% Compile error: \cq is undefined
% Expansion is disabled, and implicit token doesn't take effect.
\begingroup
\let\implicitToken\cq
\def\implicitToken{}
\cq
\endgroup


This is easier: the assignment \def\implicitToken{} overrides the \let. On the other hand, \implicitToken is never used and \cq (undefined) raises an error.

## Test 4

% test-case 4
% Output: "B"
% Expansion is enabled, but implicit token doesn't take effect.
\begingroup
\let\implicitToken=b
\edef\cq{\implicitToken}
\let\implicitToken=B
\cq
\endgroup


If you add \show\cq after doing the \edef you'll see

> \cq=macro:
->\implicitToken .


and the output will contain a “B”, because of the subsequent redefinition.

An implicit token is not expandable, so the codes

\let\implicitToken=b
\edef\cq{\implicitToken}


and

\let\implicitToken=b
\def\cq{\implicitToken}


are completely equivalent. TeX will use the meaning of \implicitToken current at expansion of \cq time: in the case of your test it is “B”.

## Test 1

% test-case 1
% Output: "b"
% Expansion is disabled, and implicit token doesn't take effect.
\begingroup
\let\implicitToken=b
\uppercase{\implicitToken}
\endgroup


The primitive \uppercase examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the \uccode vector). It doesn't act on implicit tokens.

There is an asymmetry: with \let\implicitToken=b, the test \if\implicitToken b will return true. Similarly \ifcat will use the category code of the character current at definition time; for instance

\let\implicitToken=b
\catcodeb=12
\ifcat\implicitToken a%
\message{letter}%
\else
\message{nonletter}%
\fi


will print letter on the console, whereas \ifcat ba would print nonletter.

Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with \bgroup and \egroup, which can be used as substitutes for { and } in certain (not all) contexts. I guess that making them subject to transformations in \uppercase or \lowercase would be irksome.

## Expansion

Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive \dimen is not expandable. If TeX finds \dimen2=12pt, it will pass \dimen to the execution processor; the execution prompts the search in the input stream for a <number>, an <optional equal> and a suitable value (with expansion). On the contrary, in \advance\dimen2 by 2pt, the token \dimen is absorbed as the argument to \advance for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.

What tokens are expandable? Easy:

1. some primitives, such as conditionals, \string, \noexpand and others;
2. macros, that is, tokens defined with \def, \edef, \gdef or \xdef;
3. tokens \let equal to an expandable token.

Since your \implicitToken` belongs to none of the above types, it is unexpandable.