# Tricks to make macros expandable

Expandable macros are useful (I find working in the lion's mouth super cool). But they are difficult to write.

Can more experienced users give hints that help achieve expandability?

For example, we cannot use counters since the value cannot be changed when expanding. However, a trick might be to keep the value of the counter as the number of A (say) that we move around while expanding: 5 would be 'stored' as AAAAA, and we can add counters by moving the two lists of A together, etc. Of course, it is not efficient, but it is expandable, after all.

For definiteness, say that I want to define a macro whose argument is delimited like \verb: the first character token determines what the end-character is, so that any of \foo|...|, \foo'...', \foo+...+, etc. are treated identically. Can I do this in an expandable way?

Any other trick is welcome.

-
@Bruno Would you care to expand on your statement ...one should always aim for writing expandable macros? Why? –  Yiannis Lazarides Jan 4 '11 at 16:33
@Yannis: I think what he means is macros that expand in TeX's mouth as far as possible, i.e. I would disagree with this statement in that generality, but it is sometimes a desirable, or even necessary, feature of a macro that it can do a certain amount of its work in TeX's mouth. –  Harald Hanche-Olsen Jan 4 '11 at 16:45
@Harald Thanks! There is a very good article at tug.org/TUGboat/Articles/tb11-2/tb28jeffrey.pdf all done in the mouth! –  Yiannis Lazarides Jan 4 '11 at 17:50
@Bruno. In xparse, we've explored something similar for defining expandable document commands. As you've found, there are limitations both in terms of functionality and robustness, and efficiency can be a real problem. Expandable commands are sometimes useful, but not always desirable :-) –  Joseph Wright Jan 4 '11 at 19:06
@Bruno. You'll have noticed that there is no verbatim stuff in xparse. After quite some discussion, we decided it was not a good plan. That I know of, there is no robust way to deal with cases such as \section{Text\footnote{\verb|a%c|}}, which xparse might imply. (One for LaTeX-L, if you want to discuss further.) –  Joseph Wright Jan 4 '11 at 19:15

The answer to your question about how to write expandable macros doesn't lend itself to a single correct answer, so I'll make this one CW and maybe other people will feel an urge to contribute.

• Use TeX's flexible macro argument parsing mechanism whenever possible rather than parsing input character by character (which is not expandable if you use \futurelet).
• Separate conditionals into separate macros. For example, if you want to test if an argument token is some particular token, you can use \ifx\foo#1 ...\else ...\fi, but this introduces additional tokens in the input stream. A better way to do this is to use
\def\iffoo#1{\ifx#1\foo\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi}
which will not leave any extra tokens to deal with. (Herbert wrote something similar that scooped up all text up to the \fi that was pretty clever, but I think this is clearer.) It also nests well.
• It can occasionally be useful to use a CPS.
• In several situations, the expansion of a token is the full expansion of its argument. For example, \csname ...\endcsname will expand the ... fully. This can be used to compute a string of character tokens which can be recovered, expandably, using \string:

\expandafter\expandafter\expandafter\stripslash
\expandafter\string\csname\foo\bar\baz\endcsname


This does lose the catcodes as all nonspaces will have catcode 12 and spaces will have catcode 10. In other situations the \romannumeral-X\foo trick can be used to keep expanding \foo until an unexpandable token is reached. It will swallow a space token though.

• Using ε-TeX extensions like \numexpr ...\relax, arithmetic can be performed expandably fairly easily. There is a mismatch between TeX's truncating \divide and ε-TeX's /, but this can be worked around with a trial multiplication and \ifnum.
-

Here is a partial answer to the specific challenge in this question.

\def\foo#1{\fooo#1{}}
\def\fooo#1#2#3{\ifx#1#3\unelse\bar{#2}\else\unfi\fooo#1{#2#3}\fi}
\def\unelse#1\else#2\fi{\fi#1}
\def\unfi#1\fi{\fi#1}
\def\bar#1{\message{\string\foo{#1}}}
\foo:abc:
\foo!def!
\foo|gh{i|j}|
\bye


Note how the braces in the final example shield the enclosed bar from delimiting the macro call; the braces are stripped in the process, however.

-
Nice! but note that in \foo|gh{|ij}|, the braces will not hide the vertical bar. Is it possible to read individual tokens in an expandable context using anything else than \ifx? e.g. \let and \futurelet are not expandable. –  Bruno Le Floch Jan 4 '11 at 18:06
@Bruno: Good catch! I can't see a good way around that one. I believe the answer to your question is “no”. –  Harald Hanche-Olsen Jan 4 '11 at 18:09
Yes, since we can only use \ifx or other \ifs, there is no expandable way to distinguish between |..| and |..{|} (with the standard catcodes). Every way I can think of needs duplicating the following token, and that can only be done using \let` or a macro, which has to strip braces (it would be nice to be proven wrong, though). --- By the way, inserting some conditionals inside the argument makes things break down (but that's expected). –  Bruno Le Floch Jan 4 '11 at 18:29