If you implement algorithms in TeX where variables are used, you need some kind of storage-management for storing values of variables.
At macro programming-level "values" of variables in any case are sets of tokens (control sequence tokens / explicit character tokens).
If expandability is not an issue you can use scratch-macros or scratch token-registers whose expansion yields those tokens that form the values of variables.
Each time a value of a variable is to be retrieved, expansion-trickery is to be applied for obtaining the expansion of the scratch-macro in question/\the
-expansion of the scratch-token-register in question.
Each time a variable needs to be assigned another value a TeX-assignment (\def
/\renewcommand
/\MyScratchTokenRegister={...}
/...) needs to be performed.
If expandability is an issue you can implement the algorithm in terms of tail-recursion: A macro at the end of each step of the algorithm terminates by calling itself again for performing the next step of the algorithm. The macro processes arguments, each argument denoting a variable and the tokens supplied as argument denoting the value of the variable. Each step of the algorithm is to determine the values of the variables in the next step by supplying sets of tokens as arguments for the next call of the self-calling macro. Here expansion- and flipping-around-arguments-in-the-token-stream-trickery is needed with each step of the algorithm for getting tokens/arguments arranged in the order needed by the self-calling macro, but expansion-trickery/flipping around macro-arguments usually doesn't occupy memory permanently.
An issue often neglected with expandability is that often it is handy to know about the amount of expansion-steps that need to be triggered for obtaining the result: You can have TeX trigger \romannumeral
-expansion to ensure that in any case exactly two expansion-steps need to be triggered. The first expansion-step triggers toplevel-expansion yielding a set of tokens lead by the token \romannumeral
. The second expansion-step triggers \romannumeral
. \romannumeral
in turn triggers gathering tokens belonging to the TeX-number-quantity to be represented in lowercase roman notation and hereby expanding expandable tokens. If the TeX-number-quantity to be converted has a non-positive value, \romannumeral
silently just swallows the tokens forming that TeX-number-quantity without returning any tokens. So just have your expandable tail-recursive macro collect tokens forming the result within one of its arguments. When it comes to terminating, to the tokens to be delivered by the algorithm prepend tokens that form a TeX-number-quantity of non-positive value.
Assume you wish to define a macro where 100 lines of text are accumulated so that \message
can display them.
Variant 1 - \linecollectloop
does many assignments:
\def\linescollected{}%
\def\linecollect#1#2{%
\def\linescollected{}%
\def\numberofthisline{#1}%
\def\upperbound{#2}%
\linecollectloop
}%
% macro \numberofthisline - number of current message-line
% macro \upperbound - number of last message-line to be printed
% macro \linescollected - message-lines gathered so far
\def\linecollectloop{%
\edef\linescollected{\linescollected Line \numberofthisline.^^J}%
\ifnum\numberofthisline<\upperbound\relax
\edef\numberofthisline{\number\numexpr\numberofthisline+1\relax}%
\expandafter\linecollectloop\fi
}%
\linecollect{1}{100}%
\show\linescollected
\message{\linescollected}%
\documentclass{article}
\begin{document}
\end{document}
Variant 2 - \linecollectloop
does a single assignment when terminating:
\def\fot#1#2{#1}%
\def\sot#1#2{#2}%
\def\linecollect#1#2#3{%
% #1 - number of current message-line
% #2 - number of last message-line to be printed
% #3 - message-lines gathered so far
\ifnum#1>#2 \expandafter\sot\else\expandafter\fot\fi
{\expandafter\linecollect\expandafter{\number\numexpr#1+1\relax}{#2}{#3Line #1.^^J}}%
{\def\linescollected{#3}}%
}%
\linecollect{1}{100}{}%
\show\linescollected
\message{\linescollected}%
\documentclass{article}
\begin{document}
\end{document}
Variant 3 - \linecollectloop
does no assignment at all, is fully expandable and can be combined with \romannumeral
-expansion so that you know exactly the amount of expansion-steps to be triggered via \expandafter
for obtaining the result:
\def\fot#1#2{#1}%
\def\sot#1#2{#2}%
\def\linecollect#1#2#3{%
% #1 - number of current message-line
% #2 - number of last message-line to be printed
% #3 - message-lines gathered so far; at initialization tokens
% can be prepend forming a non-postive <number>-quantity that
% terminates \romannumeral-expansion.
\ifnum#1>#2 \expandafter\sot\else\expandafter\fot\fi
{\expandafter\linecollect\expandafter{\number\numexpr#1+1\relax}{#2}{#3Line #1.^^J}}%
{#3}%
}%
\expandafter\def\expandafter\linescollected\expandafter{\romannumeral\linecollect{1}{100}{0 }}%
\show\linescollected
\message{\linescollected}%
\documentclass{article}
\begin{document}
\end{document}
\exp:w
of expl3 is \romannumeral
and \exp_end:
of expl3 is a TeX-⟨number⟩-quantity of non-positive value that terminates \exp:w
-expansion/\romannumeral
-expansion. As it is a control-word-token defined in terms of \chardef
things like \uppercase
/\lowercase
don't affect it as might be the case with a TeX-⟨number⟩-quantity that is formed by explicit character-tokens like 0⟨space⟩
.
Another issue often neglected with expandability is the handling of messages that shall be sent to console/shell/terminal or .log-file: With traditional TeX-engines one cannot trigger writing messages in an expandable way. Therefore with expandable macros/macro-mechanisms you sometimes find the practice that there is another argument where user needs to specify some tokens that shall be delivered as result at termination in case an (erroneous) situation occurred which the algorithm implemented in terms of expandable macro-mechanisms cannot handle in a way that would make sense.
On 8-bit-TeX-engines running LaTeX with inputenc-package and utf8-option loaded by default bytewise-tokenization of multi-byte-characters encoded in UTF8 often yields several character-tokens instead of a single one.
In these modern times if utf8-encoded files are processed with an 8-bit-TeX-engine underlying LaTeX, a pitfall might be the attempt of processing a sequence of characters as if each character would be represented by a single character token that can be grabbed as undelimited argument.
In the context of repeating/replicating things you often can apply \romannumeral
after appending three trailing zeros to a TeX-⟨number⟩-quantity for multiplying by 1000 and thus obtaining lowercase-letters m
in an amount corresponding to the value of the TeX-⟨number⟩-quantity in question. Then in some way or another you can have TeX iterate tail-recursively until all of these m
are consumed.
With the following example \replicate
initializes \replicateloop
by using \romannumeral
/000
-trickery for obtaining m
trailed by \relax
. \exchg
is used for placing these things as trailing items of the call to \replicateloop
.
\replicateloop
in turn processes three arguments, the first argument denoting the tokens that shall be replicated, the second argument being used for accumulating replications, the third argument either being a character m
or a control-word-token \relax
whereby \relax
denotes that no more iteration for accumulating within in the second argument shall be done but the stuff accumulated in the second argument containing replications gathered so far shall be delivered:
%define a TeX-number-quantity of non-positive value
\chardef\stopromannumeral=`\^^00
\long\def\fot#1#2{#1}%
\long\def\sot#1#2{#2}%
\long\def\exchg#1#2{#2#1}%
\long\def\replicate#1#2{%
% #1 amount of replications
% #2 tokens to replicate
\romannumeral\expandafter\exchg\expandafter{\romannumeral\number\number#1 000\relax}{\replicateloop{#2}{}}%
}%
\long\def\replicateloop#1#2#3{%
% #1 tokens to replicate
% #2 replications gathered so far
% #3 lowercase letter m or \relax
\ifx#3\relax\expandafter\sot\else\expandafter\fot\fi
{\replicateloop{#1}{#2#1}}%
{\stopromannumeral#2}%
}%
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\temp
\expandafter\expandafter\expandafter{\replicate{10}{X}}%
\message{\meaning\temp}
\csname stop\endcsname % <- this terminates in case latex is in use
\bye % <- this terrminates in case TeX is in use
\replicate
with this example delivers the result after triggering two expansion-steps:
The first expansion-step yields toplevel-expansion with a leading \romannumeral
.
The second expansion-step triggers \romannumeral
-expansion which in turn triggers everything else.
This delivers ten letters X
as message to console.
In case you don't need control over the amount of expansion-steps for obtaining the result you can do as follows:
%define a TeX-number-quantity of non-positive value
\chardef\stopromannumeral=`\^^00
\long\def\foo#1{#1}%
\long\def\noo#1{}%
\long\def\exchg#1#2{#2#1}%
\long\def\replicate#1#2{%
% #1 amount of replications
% #2 tokens to replicate
\expandafter\exchg\expandafter{\romannumeral\number\number#1 000}{\replicateloop{#2}}%
}%
\long\def\replicateloop#1#2{%
% #1 tokens to replicate
% #2 lowercase letter m or \relax
\ifx#2\relax\expandafter\noo\else\expandafter\foo\fi{#1\replicateloop{#1}}%
}%
\message{\replicate{10}{X}}
\csname stop\endcsname % <- this terminates in case latex is in use
\bye % <- this terrminates in case TeX is in use
About the \romannumeral\number\number#1 000
-construct:
I learned about this construct when looking at some code "Iterating with roman numeral" that was written by David Kastrup and is published in the Pearls 2005-section of the web presence of GUST (Polska Grupa Użytkowników Systemu TeX):
%%% David Kastrup: Iterating with roman numeral
% Appendix D in the \TeX book has the task of defining \verb+\asts+ as a
% macro containing \verb+\number\n+ copies of an asterisk. The
% solutions in the TeXbook are not really fun. Here is one that is all
% of fun, efficient and simple:
\def\asts#1{\if#1m*\expandafter\asts\fi}
\edef\asts{\expandafter\asts\romannumeral\number\n 000\relax}
% Now for something more general: we want a macro \verb+\replicate+ that
% gets a number in its first argument and arbitrary tokens in its second
% argument and expands to the given number of repeated token strings.
% It is surprisingly hard to pass \emph{both} the shrinking string of
% \verb+m+ as well as the argument to repeated in a useful way into the
% expanding first macro, and the reader is advised to try it. What I
% came up with was
\long\def\gobble#1{}
\long\def\xii#1#2{\if#2m#1\expandafter\xii\else\expandafter\gobble\fi{#1}}
\long\def\xiii#1\relax#2{\xii{#2}#1\relax}
\def\replicate#1{\expandafter\xiii\romannumeral\number\number#1 000\relax}
% A somewhat wittier variant that takes its toll on the semantic nest
% size would be
\def\recur#1{\csname rn#1\recur} \long\def\rnm#1{\endcsname{#1}#1}
\long\def\rn#1{}
\def\replicate#1{\csname rn\expandafter\recur
\romannumeral\number\number#1 000\endcsname\endcsname}
% Of course, if we are leaving the area of \TeX\ compatibility and take
% a look at what we can do with \eTeX, we arrive at the boring
\def\replicate#1#2{\ifnum#1>0 #2%
\expandafter\replicate\expandafter{\number\numexpr#1-1}{#2}\fi}
Be aware that the construct \romannumeral\number\number#1 000
works out both with TeX-⟨number⟩-quantities #1
where a trailing space is discarded during gathering and with TeX-⟨number⟩-quantities #1
where a trailing space is not discarded during gathering and with TeX-⟨number⟩-quantities #1
where a trailing space is needed. In any case \romannumeral
triggers the first \number
which in turn triggers the the second number.
E.g., with #1
= 10
you get \romannumeral\number\number10⟨space⟩000
.
The second \number
removes the space: \romannumeral\number10000
.
The first \number
just delivers the digit-sequence: \romannumeral10000
.
\romannumeral
delivers: mmmmmmmmmm
.
E.g., with \newcount\mycount \mycount=10
and #1
=\mycount
you get
\romannumeral\number\number\mycount⟨space⟩000
.
The second \number
delivers the value of \mycount
:
\romannumeral\number10⟨space⟩000
.
The first \number
removes the space: \romannumeral10000
.
\romannumeral
delivers: mmmmmmmmmm
.
E.g., with \count12=10
and #1
=\count12
you get
\romannumeral\number\number\count12⟨space⟩000
.
The second \number
delivers the value of \count12
whereby gathering the number of the register removes the space: \romannumeral\number10000
.
The first \number
just delivers the digit-sequence: \romannumeral10000
.
\romannumeral
delivers: mmmmmmmmmm
.
xparse
, we've explored something similar for defining expandable document commands. As you've found, there are limitations both in terms of functionality and robustness, and efficiency can be a real problem. Expandable commands are sometimes useful, but not always desirable :-)xparse
. After quite some discussion, we decided it was not a good plan. That I know of, there is no robust way to deal with cases such as\section{Text\footnote{\verb|a%c|}}
, whichxparse
might imply. (One for LaTeX-L, if you want to discuss further.)\def
-assignment, goes to the stomach. So the title of that article should be "Lists in TeX's Gullet" rather than "Lists in TeX's Mouth". But that doesn't sound nice.