How does your mind bend \expandafter to its will?

Question

Expansion is often cited as one of the most arcane aspects of TeX, more akin to witchcraft than to something easily picked up by the newcomer. There are many great questions and great answers about expansion on the site but, although I like to think I'm getting better at it, I still find myself stumped by more complicated cases.

For instance, take the LaTeX internal called \in@. Martin Scharrer's excellent List of internal LaTeX2e macros explains \in@ as follows:

\in@{⟨1⟩}{⟨2⟩}

Checks if first argument occurs in the second and sets the switch \ifin@ accordantly. The arguments are not expanded. This must be done beforehand.

Because I'd like to pass once-expandable macros as arguments to \in@, I need to expand each of those arguments once before \in@ gets expanded.

I managed to handle the case where the first argument needs to be expanded once and the second argument doesn't need expansion (see my code below).

However, I can't get my head around the \expandafter juggling required to handle the case in which both arguments need to be expanded once. I think (correct me if I'm wrong) that I have to leave the braces untouched until \in@ itself is processed, and that's what I find difficult. I'd appreciate your help on this particular expansion problem.

\documentclass{article}

\let\ex=\expandafter % <--- more readable \expandafter

\begin{document}
\makeatletter

\in@{foo}{foo,bar,baz} % <--- works as expected
\ifin@ true\else false\fi

\def\foo{foo}
\ex\in@\ex{\foo}{foo,bar,baz} % <--- so far, so good...
\ifin@ true\else false\fi

\def\cslist{foo,bar,baz}
\in@{\foo}{\cslist} % <--- What combination of \expandafter is needed here?
\ifin@ true\else false\fi

\makeatother
\end{document}

However, more generally, I'd like to know what general procedure expansion gurus follow in their head (and possibly on paper) to expand a sequence of tokens in the desired order. Stephan v. Bechtolsheim's A tutorial on \expandafter gave me some insight, but I'm still far from mastering it. I think sharing your tricks and recipes might help me and others improve our expansion skills.

So, what cognitive processes allow you to bend \expandafter to your will?

Note: I insist on using only \expandafter here, and none of that \edef / \noexpand trickery. :)

Answer 1

\in@{\foo}{\cslist} % <--- What combination of \expandafter is needed here?

If \foo is first expanded, then we have the problem, that \expandafter cannot jump over serveral tokens at once, also the number of tokens is not known. Therefore the latest token is expanded first. But at this stage we cannot add the \expandafter, because we have to insert the \expandafter tokens for \foo first:

\ex\in@\ex{\foo}{\cslist}

Then we add the outmost \expandafter chain to expand \cslist. The next line uses \EX for the new \expandafter to make the difference between the stages visible:

\EX\ex\EX\in@\EX\ex\EX{\EX\foo\EX}\EX{\cslist}

Result:

\ex\ex\ex\in@\ex\ex\ex{\ex\foo\ex}\ex{\cslist}

A more generic algorithm would be:

1. Establish an order of expansions:

• Collapsing expansions: We have to make sure, that the number of tokens is known and we can insert \expandafter between them, if we need to jump over them. Therefore constructs like \csname need to be expanded at an earlier level. This allows also the use of arguments inside \csname with an unknown number of tokens, because the \csname construct becomes one single command token after one expansion step.

  Note: See also the trick below, that `\csname` can be used to expand stuff afterwards.
  

• Expanding expansions, e.g. \cslist above, on the right side have to come first, because we cannot jump over a unknown number of tokens.

2. Now we can add \expandafter chains from the start to the token that needs expanding. The order from the previous step is now reversed. First the chain for the token that is last expanded is inserted, e.g.:

    0. \a\b\last\c\first
    1. \EX\a\EX\b\last\c\first % \EX inserted
   

Then we go backwards in time to expand the token that needs expansion before the last:

    =1. \ex\a\ex\b\last\c\first
     2. \EX\ex\EX\a\EX\ex\EX\b\EX\last\EX\c\first % \EX inserted
    =2. \ex\ex\ex\a\ex\ex\ex\b\ex\last\ex\c\first

"Tricks"

Sometimes TeX helps to save some \expandafter.

• Expanding after \csname:

  Let's assume \foo and \cslist are not given explicitly but constructed via \csname:

    \in@{\csname foo\endcsname}{\csname cslist\endcsname}
  

A naive approach would require four expansions waves:

1. expanding the first \csname to get one token \foo
2. expanding the second \csname to get \cslist
3. expanding \cslist
4. expanding \foo

Result: Start with 2⁴-1 \expandafter (= 15).

This can be reduced: TeX expands the tokens between \csname and \endcsname until nothing expandable is left to form a command sequence. The following uses this to get \cslist and its expansion before \foo is constructed:

    \csname foo\ex\ex\ex\endcsname
    \ex\ex\ex}\ex\ex\ex{\csname cslist\endcsname}

And the whole expression with the expansion of \foo:

    \ex\ex\ex\in@\ex\ex\ex{\csname foo\ex\ex\ex\endcsname
    \ex\ex\ex}\ex\ex\ex{\csname cslist\endcsname}

The result are 15 \expandafter in total.

• Expanding arguments of some TeX primitives such as \uppercase.

  Let's assume \foo expands to a word that should be converted to uppercase:

    \ex\uppercase\ex{\foo}
  

  Here we can save the first \expandafter, because \uppercase already expands the next tokens until it gets the opening brace:

    \uppercase\ex{\foo}
  

  Other primitives: \detokenize, \scantokens, \message.

  Caveat: If someone redefines \uppercase as macro, this trick will fail obviously.

Answer 2

You can exploit \unexpanded:

\documentclass{article}

\makeatletter
% both arguments are expanded once
\newcommand{\xxin@}[2]{%
  \begingroup\edef\x{\endgroup
    \noexpand\in@{\unexpanded\expandafter{#1}}{\unexpanded\expandafter{#2}}%
  }\x
}
% the first argument is expanded once
\newcommand{\xnin@}[2]{%
  \begingroup\edef\x{\endgroup
    \noexpand\in@{\unexpanded\expandafter{#1}}{\unexpanded{#2}}%
  }\x
}
% the second argument is expanded once
\newcommand{\nxin@}[2]{%
  \begingroup\edef\x{\endgroup
    \noexpand\in@{\unexpanded{#1}}{\unexpanded\expandafter{#2}}%
  }\x
}
\makeatletter

\begin{document}
\makeatletter
\def\foo{foo}
\def\cslist{foo,bar,baz}

\in@{foo}{foo,bar,baz}
\ifin@ true\else false\fi

\xnin@{\foo}{foo,bar,baz}
\ifin@ true\else false\fi

\nxin@{foo}{\cslist}
\ifin@ true\else false\fi

\xxin@{\foo}{\cslist}
\ifin@ true\else false\fi

\makeatother
\end{document}

This prints “true” for all cases.

---

The mandatory LaTeX3 solution, where \tl_if_in:nnTF (with variants) is provided.

\documentclass{article}
\usepackage{expl3}
\ExplSyntaxOn
\cs_generate_variant:Nn \tl_if_in:nnTF { oo }
\ExplSyntaxOff

\begin{document}

\def\foo{foo}
\def\cslist{foo,bar,baz}

\ExplSyntaxOn
\tl_if_in:nnTF{foo,bar,baz}{foo}{true}{false}\par

\tl_if_in:noTF{foo,bar,baz}{\foo}{true}{false}\par

\tl_if_in:onTF{\cslist}{foo}{true}{false}\par

\tl_if_in:ooTF{\cslist}{\foo}{true}{false}\par
\ExplSyntaxOff
\end{document}

Answer 3

\in@{\foo}{\cslist}

If you don't need pure expansion and can afford an assignment, that can simplify things

\def\tmp{\expandafter\in@\expandafter{\foo}}
\expandafter\tmp\expandafter{\cslist}

only needs four \expandafter

Answer 4

For the sake of the exercise, here is a method without assignments. Don't expect miracles though, it should be fine when testing strings of letters and digits, however spaces will give false positives. This is a first sketch, I use xinttools as I am familiar with it.

Some explanations added.

\documentclass{article}
\usepackage{xinttools}

\makeatletter
% This takes care of expanding once the first argument #1, and also \detokenize
% it. 

% regarding #2, the list argument:
% the macro \xintCSVtoList expands repeatedly the first token of its argument
% by a method which stops when a space token is encountered. Thus, we use \expandafter\space
% in order for #2 to be expanded only once. Then tne comma separated list is converted 
% to a sequence of braced items (this is called in List by xint).

\def\IFIN #1#2{\expandafter\xintApplyUnbraced\expandafter
              {\expandafter\IFIN@a\expandafter{\detokenize\expandafter{#1}}}%
              {\xintCSVtoList{\expandafter\space#2}}\@secondoftwo}

% once #1 and #2 have each been expanded once, each item from the list will
% be given as (second) argument to \IFIN@a{alreaedy expanded #1}
% Thus the macro \IFIN@a has two arguments which it wants to compare. But this
% must be done expandably. First, let's use \detokenize to simplify things, 
% then we will compare token by token.

% update: the \detokenize {#1} was done already. Only remains the list item #2.

% \def\IFIN@a #1{\expandafter\IFIN@b\expandafter{\detokenize{#1}}}

\def\IFIN@a #1#2{\expandafter\IFIN@c\detokenize{#2}\dummy #1\dummy}

% if this #1 (which was originally an item from the comma separated list) is
% empty, the tested thing should rather be empty too.

% thus, \IFIN@x will test that emptiness, else, we go to \IFIN@d
\def\IFIN@c #1\dummy {\ifx\relax#1\relax\expandafter\IFIN@x \else
                         \expandafter\IFIN@d \fi {#1}}

% we know #1 is not empty, if #2 is empty, check is done, if not we go to \IFIN@e
% to compare the first two tokens.

\def\IFIN@d #1#2\dummy {\ifx\relax#2\relax\expandafter\IFIN@z\else
                           \expandafter\IFIN@e\fi #1\dummy #2\dummy }

% the core test; as the non delimited #1 and #3 will never be a space token, this means
% spaces create false positives (\detokenize gives catcode 12 to everything but spaces)

\def\IFIN@e #1#2\dummy #3#4\dummy {\ifx #1#3\expandafter\IFIN@c\else
                                    \expandafter\IFIN@z\fi #2\dummy
                                    #4\dummy}

% why the \space? because \xintApplyUnbraced tries to fully expand each 
% application of the macro \IFIN@a {#1}, the \space will stop that. This is
% for clarity of code as anyhow if one looks at the source code of \xintApplyUnbraced,
% one sees the expansion will be stopped by a closing brace.

\def\IFIN@x #1#2\dummy {\ifx\relax#2\relax\expandafter\IFIN@YES\else
                        \expandafter\space\fi}

% same business with a space. This is optional.
\def\IFIN@z #1\dummy#2\dummy { }

% here the space is not optional, we do not want premature expansion of \IFIN@@YES

\def\IFIN@YES { \IFIN@@YES}

\def\IFIN@@YES #1\@secondoftwo {\@firstoftwo}

% the \xintApplyUnbraced needs two expansion steps to do its job, which will
% produce either \IFIN@@YES...\IFIN@@YES, if there was one or more
% matches or nothing at all if there was no match. Thus the case of a match 
% needs two more expansions to choose the branch, the case of a mismatch
% only one more expansion.

% It is possible to modify the code in order for \IFIN to, in exactly two steps
% choose the YES or NO branch. Here is how to:
% \def\IFIN #1#2{\romannumeral0\expandafter\xintApplyUnbraced\expandafter
%               {\expandafter\IFIN@a\expandafter{\detokenize\expandafter{#1}}}%
%               {\xintCSVtoList{#2}}\expandafter\space\@secondoftwo}
% \def\IFIN@@YES #1\@secondoftwo {\expandafter\space\@firstoftwo}
% One can check that the \test macros below expand in only two steps
% through using \oodef (provided by xinttools) rather than \edef.    


\begin{document}\thispagestyle{empty}
\ttfamily
\def\cslist{foo,bar,baz}
\meaning\cslist

%
\def\foo {bar}

\IFIN\foo\cslist {bar in list}{bar not in list}

\edef\test {\IFIN\foo\cslist {bar in list}{bar not in list}}
% use \oodef\test, if \IFIN and \IFIN@@YES are modified as explained in the 
% comments above

\meaning\test
%--

\def\foo{Bar}

\IFIN\foo\cslist {Bar in list}{Bar not in list}

\edef\test {\IFIN\foo\cslist {Bar in list}{Bar not in list}}

\meaning\test

%--
\def\foo {foo}

\IFIN\foo\cslist {foo in list}{foo not in list}

\edef\test {\IFIN\foo\cslist {foo in list}{foo not in list}}

\meaning\test
%--

\def\foo{Foo}

\IFIN\foo\cslist {Foo in list}{Foo not in list}

\edef\test {\IFIN\foo\cslist {Foo in list}{Foo not in list}}

\meaning\test

%--

\def\foo {baz}

\IFIN\foo\cslist {baz in list}{baz not in list}

\edef\test {\IFIN\foo\cslist {baz in list}{baz not in list}}

\meaning\test

%--

\def\foo{Baz}

\IFIN\foo\cslist {Baz in list}{Baz not in list}

\edef\test {\IFIN\foo\cslist {Baz in list}{Baz not in list}}

\meaning\test

%--

\def\foo{document}

\IFIN\foo\cslist {document in list}{document not in list}

\edef\test {\IFIN\foo\cslist {document in list}{document not in list}}

\meaning\test

%--

\def\foo{1}

\IFIN\foo\cslist {1 in list}{1 not in list}

\edef\test {\IFIN\foo\cslist {1 in list}{1 not in list}}

\meaning\test

%--

\def\foo{b a r}

\IFIN\foo\cslist {b a r in list}{b a r not in list}

\edef\test {\IFIN\foo\cslist {b a r in list}{b a r not in list}}

\meaning\test

\end{document}

Answer 5

I write some code to generate such expandafter chains.

If you compile the following code...

%! TEX program = lualatex
\documentclass{article}
\usepackage{prettytok}
\ExplSyntaxOn \prettyinit: \ExplSyntaxOff
\usepackage{imperative}
\AutoGenerateVariants
\errorcontextlines=100
\begin{document}

\ExplSyntaxOn
\makeatletter

\debug_putnextwithexpand {
    \in@ {\foo} {\cslist}
} {
    \expandat \cslist
    \expandat \foo
}


\debug_putnextwithexpand {
    \in@{ ! \csname foo * \endcsname}{ ^ \csname cslist\endcsname}
} {
    \expandatlabel !
    \expandatlabel !
    \onlabel * \expandatlabel ^
    \onlabel * \expandatlabel ^
}

\debug_putnextwithexpand {
    \uppercase { \foo }
} {
    \after \uppercase \expandat \foo
}

\ExplSyntaxOff
\makeatother

\end{document}

you'll get the resulting code that can be found in the answer above:

The "compiler" itself is LuaLaTeX-only, nevertheless the "compiled code" can be used in any compiler.

You do need to download some .sty files from https://github.com/user202729/TeXlib to actually run the code however.

(I hope the code is self-explanatory even without documentation...)

Side note, there are two less \ex in the first piece of code, but my version appears to be correct according to manual inspection.