Tricks to make macros expandable

Question

Expandable macros are useful (I find working in the lion's mouth super cool). But they are difficult to write.

Can more experienced users give hints that help achieve expandability?

For example, we cannot use counters since the value cannot be changed when expanding. However, a trick might be to keep the value of the counter as the number of A (say) that we move around while expanding: 5 would be 'stored' as AAAAA, and we can add counters by moving the two lists of A together, etc. Of course, it is not efficient, but it is expandable, after all.

For definiteness, say that I want to define a macro whose argument is delimited like \verb: the first character token determines what the end-character is, so that any of \foo|...|, \foo'...', \foo+...+, etc. are treated identically. Can I do this in an expandable way?

Any other trick is welcome.

Answer 1

The answer to your question about how to write expandable macros doesn't lend itself to a single correct answer, so I'll make this one CW and maybe other people will feel an urge to contribute.

• Use TeX's flexible macro argument parsing mechanism whenever possible rather than parsing input character by character (which is not expandable if you use \futurelet).

• Separate conditionals into separate macros. For example, if you want to test if an argument token is some particular token, you can use \ifx\foo#1 ...\else ...\fi, but this introduces additional tokens in the input stream. A better way to do this is to use \def\iffoo#1{\ifx#1\foo\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi} which will not leave any extra tokens to deal with. (Herbert wrote something similar that scooped up all text up to the \fi that was pretty clever, but I think this is clearer.) It also nests well.

• It can occasionally be useful to use a CPS.

• In several situations, the expansion of a token is the full expansion of its argument. For example, \csname ...\endcsname will expand the ... fully. This can be used to compute a string of character tokens which can be recovered, expandably, using \string:

   \expandafter\expandafter\expandafter\stripslash
       \expandafter\string\csname\foo\bar\baz\endcsname
  

  This does lose the catcodes as all nonspaces will have catcode 12 and spaces will have catcode 10. In other situations the \romannumeral-`X\foo trick can be used to keep expanding \foo until an unexpandable token is reached. It will swallow a space token though.

• Using ε-TeX extensions like \numexpr ...\relax, arithmetic can be performed expandably fairly easily. There is a mismatch between TeX's truncating \divide and ε-TeX's /, but this can be worked around with a trial multiplication and \ifnum.

Answer 2

Here is a partial answer to the specific challenge in this question.

\def\foo#1{\fooo#1{}}
\def\fooo#1#2#3{\ifx#1#3\unelse\bar{#2}\else\unfi\fooo#1{#2#3}\fi}
\def\unelse#1\else#2\fi{\fi#1}
\def\unfi#1\fi{\fi#1}
\def\bar#1{\message{\string\foo{#1}}}
\foo:abc:
\foo!def!
\foo|gh{i|j}|
\bye

Note how the braces in the final example shield the enclosed bar from delimiting the macro call; the braces are stripped in the process, however.

Answer 3

If you implement algorithms in TeX where variables are used, you need some kind of storage-management for storing values of variables.

At macro programming-level "values" of variables in any case are sets of tokens (control sequence tokens / explicit character tokens).

If expandability is not an issue you can use scratch-macros or scratch token-registers whose expansion yields those tokens that form the values of variables. Each time a value of a variable is to be retrieved, expansion-trickery is to be applied for obtaining the expansion of the scratch-macro in question/\the-expansion of the scratch-token-register in question. Each time a variable needs to be assigned another value a TeX-assignment (\def/\renewcommand/\MyScratchTokenRegister={...}/...) needs to be performed.

If expandability is an issue you can implement the algorithm in terms of tail-recursion: A macro at the end of each step of the algorithm terminates by calling itself again for performing the next step of the algorithm. The macro processes arguments, each argument denoting a variable and the tokens supplied as argument denoting the value of the variable. Each step of the algorithm is to determine the values of the variables in the next step by supplying sets of tokens as arguments for the next call of the self-calling macro. Here expansion- and flipping-around-arguments-in-the-token-stream-trickery is needed with each step of the algorithm for getting tokens/arguments arranged in the order needed by the self-calling macro, but expansion-trickery/flipping around macro-arguments usually doesn't occupy memory permanently.

An issue often neglected with expandability is that often it is handy to know about the amount of expansion-steps that need to be triggered for obtaining the result: You can have TeX trigger \romannumeral-expansion to ensure that in any case exactly two expansion-steps need to be triggered. The first expansion-step triggers toplevel-expansion yielding a set of tokens lead by the token \romannumeral. The second expansion-step triggers \romannumeral. \romannumeral in turn triggers gathering tokens belonging to the TeX-number-quantity to be represented in lowercase roman notation and hereby expanding expandable tokens. If the TeX-number-quantity to be converted has a non-positive value, \romannumeral silently just swallows the tokens forming that TeX-number-quantity without returning any tokens. So just have your expandable tail-recursive macro collect tokens forming the result within one of its arguments. When it comes to terminating, to the tokens to be delivered by the algorithm prepend tokens that form a TeX-number-quantity of non-positive value.

Assume you wish to define a macro where 100 lines of text are accumulated so that \message can display them.

Variant 1 - \linecollectloop does many assignments:

\def\linescollected{}%
\def\linecollect#1#2{%
  \def\linescollected{}%
  \def\numberofthisline{#1}%
  \def\upperbound{#2}%
  \linecollectloop
}%
% macro \numberofthisline - number of current message-line
% macro \upperbound - number of last message-line to be printed
% macro \linescollected - message-lines gathered so far
\def\linecollectloop{%
  \edef\linescollected{\linescollected Line \numberofthisline.^^J}%
  \ifnum\numberofthisline<\upperbound\relax
    \edef\numberofthisline{\number\numexpr\numberofthisline+1\relax}%
    \expandafter\linecollectloop\fi
}%
\linecollect{1}{100}%
\show\linescollected
\message{\linescollected}%
\documentclass{article}
\begin{document}
\end{document}

Variant 2 - \linecollectloop does a single assignment when terminating:

\def\fot#1#2{#1}%
\def\sot#1#2{#2}%
\def\linecollect#1#2#3{%
  % #1 - number of current message-line
  % #2 - number of last message-line to be printed
  % #3 - message-lines gathered so far
  \ifnum#1>#2 \expandafter\sot\else\expandafter\fot\fi
  {\expandafter\linecollect\expandafter{\number\numexpr#1+1\relax}{#2}{#3Line #1.^^J}}%
  {\def\linescollected{#3}}%
}%
\linecollect{1}{100}{}%
\show\linescollected
\message{\linescollected}%
\documentclass{article}
\begin{document}
\end{document}

Variant 3 - \linecollectloop does no assignment at all, is fully expandable and can be combined with \romannumeral-expansion so that you know exactly the amount of expansion-steps to be triggered via \expandafter for obtaining the result:

\def\fot#1#2{#1}%
\def\sot#1#2{#2}%
\def\linecollect#1#2#3{%
  % #1 - number of current message-line
  % #2 - number of last message-line to be printed
  % #3 - message-lines gathered so far; at initialization tokens
  %      can be prepend forming a non-postive <number>-quantity that
  %      terminates \romannumeral-expansion.
  \ifnum#1>#2 \expandafter\sot\else\expandafter\fot\fi
  {\expandafter\linecollect\expandafter{\number\numexpr#1+1\relax}{#2}{#3Line #1.^^J}}%
  {#3}%
}%
\expandafter\def\expandafter\linescollected\expandafter{\romannumeral\linecollect{1}{100}{0 }}%
\show\linescollected
\message{\linescollected}%
\documentclass{article}
\begin{document}
\end{document}

\exp:w of expl3 is \romannumeral and \exp_end: of expl3 is a TeX-⟨number⟩-quantity of non-positive value that terminates \exp:w-expansion/\romannumeral-expansion. As it is a control-word-token defined in terms of \chardef things like \uppercase/\lowercase don't affect it as might be the case with a TeX-⟨number⟩-quantity that is formed by explicit character-tokens like 0⟨space⟩.

Another issue often neglected with expandability is the handling of messages that shall be sent to console/shell/terminal or .log-file: With traditional TeX-engines one cannot trigger writing messages in an expandable way. Therefore with expandable macros/macro-mechanisms you sometimes find the practice that there is another argument where user needs to specify some tokens that shall be delivered as result at termination in case an (erroneous) situation occurred which the algorithm implemented in terms of expandable macro-mechanisms cannot handle in a way that would make sense.

---

On 8-bit-TeX-engines running LaTeX with inputenc-package and utf8-option loaded by default bytewise-tokenization of multi-byte-characters encoded in UTF8 often yields several character-tokens instead of a single one.

In these modern times if utf8-encoded files are processed with an 8-bit-TeX-engine underlying LaTeX, a pitfall might be the attempt of processing a sequence of characters as if each character would be represented by a single character token that can be grabbed as undelimited argument.

---

In the context of repeating/replicating things you often can apply \romannumeral after appending three trailing zeros to a TeX-⟨number⟩-quantity for multiplying by 1000 and thus obtaining lowercase-letters m in an amount corresponding to the value of the TeX-⟨number⟩-quantity in question. Then in some way or another you can have TeX iterate tail-recursively until all of these m are consumed.

With the following example \replicate initializes \replicateloop by using \romannumeral/000-trickery for obtaining m trailed by \relax. \exchg is used for placing these things as trailing items of the call to \replicateloop. \replicateloop in turn processes three arguments, the first argument denoting the tokens that shall be replicated, the second argument being used for accumulating replications, the third argument either being a character m or a control-word-token \relax whereby \relax denotes that no more iteration for accumulating within in the second argument shall be done but the stuff accumulated in the second argument containing replications gathered so far shall be delivered:

%define a TeX-number-quantity of non-positive value
\chardef\stopromannumeral=`\^^00
\long\def\fot#1#2{#1}%
\long\def\sot#1#2{#2}%
\long\def\exchg#1#2{#2#1}%
\long\def\replicate#1#2{%
  % #1 amount of replications
  % #2 tokens to replicate
  \romannumeral\expandafter\exchg\expandafter{\romannumeral\number\number#1 000\relax}{\replicateloop{#2}{}}%
}%
\long\def\replicateloop#1#2#3{%
  % #1 tokens to replicate
  % #2 replications gathered so far
  % #3 lowercase letter m or \relax
  \ifx#3\relax\expandafter\sot\else\expandafter\fot\fi
  {\replicateloop{#1}{#2#1}}%
  {\stopromannumeral#2}%
}%
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\temp
\expandafter\expandafter\expandafter{\replicate{10}{X}}%
\message{\meaning\temp}
\csname stop\endcsname % <- this terminates in case latex is in use
\bye % <- this terrminates in case TeX is in use

\replicate with this example delivers the result after triggering two expansion-steps:
The first expansion-step yields toplevel-expansion with a leading \romannumeral.
The second expansion-step triggers \romannumeral-expansion which in turn triggers everything else.

This delivers ten letters X as message to console.

In case you don't need control over the amount of expansion-steps for obtaining the result you can do as follows:

%define a TeX-number-quantity of non-positive value
\chardef\stopromannumeral=`\^^00
\long\def\foo#1{#1}%
\long\def\noo#1{}%
\long\def\exchg#1#2{#2#1}%
\long\def\replicate#1#2{%
  % #1 amount of replications
  % #2 tokens to replicate
  \expandafter\exchg\expandafter{\romannumeral\number\number#1 000}{\replicateloop{#2}}%
}%
\long\def\replicateloop#1#2{%
  % #1 tokens to replicate
  % #2 lowercase letter m or \relax
  \ifx#2\relax\expandafter\noo\else\expandafter\foo\fi{#1\replicateloop{#1}}%
}%
\message{\replicate{10}{X}}
\csname stop\endcsname % <- this terminates in case latex is in use
\bye % <- this terrminates in case TeX is in use

About the \romannumeral\number\number#1 000-construct:

I learned about this construct when looking at some code "Iterating with roman numeral" that was written by David Kastrup and is published in the Pearls 2005-section of the web presence of GUST (Polska Grupa Użytkowników Systemu TeX):

%%% David Kastrup: Iterating with roman numeral
% Appendix D in the \TeX book has the task of defining \verb+\asts+ as a
% macro containing \verb+\number\n+ copies of an asterisk.  The
% solutions in the TeXbook are not really fun.  Here is one that is all
% of fun, efficient and simple:

\def\asts#1{\if#1m*\expandafter\asts\fi}
\edef\asts{\expandafter\asts\romannumeral\number\n 000\relax}

% Now for something more general: we want a macro \verb+\replicate+ that
% gets a number in its first argument and arbitrary tokens in its second
% argument and expands to the given number of repeated token strings.

% It is surprisingly hard to pass \emph{both} the shrinking string of
% \verb+m+ as well as the argument to repeated in a useful way into the
% expanding first macro, and the reader is advised to try it.  What I
% came up with was

\long\def\gobble#1{}
\long\def\xii#1#2{\if#2m#1\expandafter\xii\else\expandafter\gobble\fi{#1}}
\long\def\xiii#1\relax#2{\xii{#2}#1\relax}
\def\replicate#1{\expandafter\xiii\romannumeral\number\number#1 000\relax}

% A somewhat wittier variant that takes its toll on the semantic nest
% size would be

\def\recur#1{\csname rn#1\recur} \long\def\rnm#1{\endcsname{#1}#1}
\long\def\rn#1{}
\def\replicate#1{\csname rn\expandafter\recur
  \romannumeral\number\number#1 000\endcsname\endcsname}

% Of course, if we are leaving the area of \TeX\ compatibility and take
% a look at what we can do with \eTeX, we arrive at the boring

\def\replicate#1#2{\ifnum#1>0 #2%
  \expandafter\replicate\expandafter{\number\numexpr#1-1}{#2}\fi}

Be aware that the construct \romannumeral\number\number#1 000 works out both with TeX-⟨number⟩-quantities #1 where a trailing space is discarded during gathering and with TeX-⟨number⟩-quantities #1 where a trailing space is not discarded during gathering and with TeX-⟨number⟩-quantities #1 where a trailing space is needed. In any case \romannumeral triggers the first \number which in turn triggers the the second number.

E.g., with #1 = 10 you get \romannumeral\number\number10⟨space⟩000.
The second \number removes the space: \romannumeral\number10000.
The first \number just delivers the digit-sequence: \romannumeral10000.
\romannumeral delivers: mmmmmmmmmm.

E.g., with \newcount\mycount \mycount=10 and #1=\mycount you get
\romannumeral\number\number\mycount⟨space⟩000.
The second \number delivers the value of \mycount:
\romannumeral\number10⟨space⟩000.
The first \number removes the space: \romannumeral10000.
\romannumeral delivers: mmmmmmmmmm.

E.g., with \count12=10 and #1=\count12 you get
\romannumeral\number\number\count12⟨space⟩000.
The second \number delivers the value of \count12 whereby gathering the number of the register removes the space: \romannumeral\number10000.
The first \number just delivers the digit-sequence: \romannumeral10000.
\romannumeral delivers: mmmmmmmmmm.