TeX macro idioms, or: understanding advanced macros

Question

I often find that when I look at solutions from the experts here, the macro definitions are completely inscrutable to me. Trying to use standard techniques one would use to understand code in most programming languages often isn't helpful; for example, I find that chasing definitions using show can bottom out in something that still doesn't make sense (to me).

Now, @David Carlisle has just very kindly taught me a TeX idiom which suddenly makes a lot of macros make sense. This question is largely motivated by a desire to share that with other people at my level. It's asking for descriptions of any common idioms that:

1. occur frequently in packages and answers from experts here
2. are doing something that isn't obvious from just understanding the commands. (For example, lccode used to something that has nothing to do with lowercase characters.)

The idiom I've just learnt is going into an answer below, and should help illustrate what I mean.

Answer 1

One of the more frustrating features of TeX's macro-expansion based language is that the control code and the code it controls can interact unexpectedly. This means that if you want a conditional block \ifwhatever...\fi to control some macro \macro that affects stuff after the block, you need to leapfrog \macro to finish the conditional first:

\ifwhatever
 \expandafter\macro
\fi
<stuff taken as an argument by \macro>

You see, the \expandafter eats the \fi before \macro can. Using this trick, you can write some code that conditionally executes some following code. Say you have a conditional \ifwhatever, and you want to execute <code> if it is true, but not if it is false. Then you can write (here I am assuming \makeatletter is active):

\ifwhatever
 \expandafter\@firstofone
\else
 \expandafter\@gobble
\fi
{<code>}

This can be generalized if you want to select among two different pieces of code, which by the way is how the LaTeX-style conditionals \ifwhatever{<true code>}{<false code>} work:

\ifwhatever
 \expandafter\@firstoftwo
\else
 \expandafter\@secondoftwo
\fi
{<code 1>}
{<code 2>}

I wouldn't advise doing this with more than one conditional, though: if you want to execute <code> when \ifwhatever or \ifnot is true, you have to write:

\ifwhatever
 \expandafter\@gobble
\else
 \ifnot
   \expandafter\expandafter\expandafter\@firstoftwo
 \else
   \expandafter\expandafter\expandafter\@gobble
 \fi
\fi
{<code>}

This profusion of \expandafters is to skip \@firstoftwo or \@gobble and expand first one token, then another after it, before going back to expand the macro you actually want once there is nothing in its way.

Of course, the technique of using multiple \expandafters to skip ahead is itself an idiom. For example, you can expand the first token in a group once:

\expandafter{\macro}

or twice:

\expandafter\expandafter
\expandafter{\macro}

or three times:

\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter
\expandafter{\macro}

and you see the pattern. I may as well explain why this works. When the first \expandafter goes off, it expands in turn the third, then the first in the next row, then the first in the following row, and finally, \macro. The intervening tokens are not expanded in this "run", but just stuck back in order. The effect is that every other column of \expandafters vanishes, \macro is expanded once, and then TeX starts expanding again with what is now the first \expandater, which was the second. The configuration has become the one for expanding \macro twice, and so we end up expanding it three times.

Answer 2

lccode + lowercase trick

@David Carlisle's trick is used when you want to redefine the behaviour of a standard (say) ASCII character, such as - or \n. This is useful in e.g. verbatim-like environments, where you might want a newline to be processed in a non-standard way -- and I'm going to stick with that example for the rest of this post.

TeX lets you make any such character into a macro using \catcode and \active, which gives you a easy way of changing the behaviour of \n. The problem is that as soon as you've done that, then every newline in your source code will start behaving differently, which makes it rather hard to actually write anything. So you need some way of being able to use 'inactive' newlines in your source while treating any newlines typed by your user as 'active'.

As I understand it, the trick works as follows:

1. You pick an arbitrary active character which you don't need to use in your macro. ~ is a good choice because it is always active. If you need more, you make whatever characters you like active.

2. You "tell" TeX that the lowercase version of (say) ~ is \n, like this:

   \lccode`~=`\^^M 
   

   This is of course nothing to do with actual lowercase text.

3. You wrap your entire macro in \lowercase{...}. This means that whenever you use ~, TeX converts it into \n. And because ~ is active, it converts it into an active \n. This means, in particular, that you can redefine the behaviour of \n like this:

   \lowercase{
   ...
   \def~{...}
   ...
   }
   

   At the same time, normal newlines in your macro (like the one before the final }) remain inactive, and so are processed by TeX in the normal fashion.

[At least, that's it as I best understand it... ]

Answer 3

I suppose we should have mention of

{\ifnum0=`}\fi 

\ifnum0=`{\fi}

The brace group constructs that are endemic in packages related to tabular and related alignment constructs.

To understand these constructs it is best to start in the middle: the backtick construct `{ returns the character code of { which is 123 and in particular it is not 0 so

\ifnum0=`}\fi 

is like \iffalse\fi and expands to nothing. Similarly

\ifnum0=`{\fi

Thus the first construct expands to an explicit { and the second construct expands to an explicit }

there are simpler constructs that expand to explicit braces notably

{\iffalse}\fi

\iffalse{\fi}

However as we shall see they do not work in the tabular constructs the way that is needed.

Now consider a simple halign construct (just using plain TeX for simplicity) that defines centred columns and (like LaTeX) wants to use a local definition of \\ to end the row.

\def\tabc{%
\leavevmode
\vtop\bgroup\let\\\cr
\halign\bgroup&\quad\hfil\ignorespaces##\unskip\hfil\quad\cr}

\def\endtabc{%
\crcr\egroup
\egroup}



\tabc
1 & 2 & 3\\
aaa & bbb & ccc
\endtabc



\bye

This apparently works fine and produces

Then someone decides that they want to nest tables …

Replacing the 1 in the first cell by a nested 2x2 table:

\tabc
\tabc a & b \cr x & y \endtabc & 2 & 3\\
aaa & bbb & ccc
\endtabc

This fails with the error message

! Missing } inserted.
<inserted text> 
                }
<to be read again> 
                   \endtemplate 
<template> \unskip \hfil \quad \endtemplate 

\tabc ->\leavevmode \vtop \bgroup \let \\
                                         \cr \halign \bgroup &\quad \hfil \i...
l.14 \tabc
           a & b \cr x & y \endtabc & 2 & 3The hint to understanding the error message is the `\\ ` at the end of the line has cause `\endtemplate` to be read. What has happened is that the intended meaning of locally defining `\\ ` has not happened: as soon as TeX saw the `\\ ` (which is `\let` to `\cr` already because of the outer table, the cell of the outer table ended, the the `\let` then defined `\unskip` to be `\hfil`, the `quad` made a space, but then the cell ended in while the group started by `\halign\bgroup` is still open, so an error was generated.

One work-around is to nest the inner table in explicit braces:

\tabc
{\tabc a & b \cr x & y \endtabc} & 2 & 3\\
aaa & bbb & ccc
\endtabc

works as intended and produces

Sometimes you see table macros with similar requirements that nested instances be protected by explicit braces. But normally it is better to make the macros safe for nested use.

Due to the way that macro definitions are parsed, it is not possible to simply add a { to the definition of \tabc. Also the usual implicit brace \bgroup does not work here, note that the redefinition of \\ is already inside a group started by \bgroup but TeX's table cell scanner does not see that in the right way. It turns out that the construct at the start is exactly what is required. If we define the table macro as below then the nested table works as expected without the need for extra braces in the document: Once the inner \tabc is expanded in the first cell of the outer table the {\ifnum0=`}\fi is expanded and TeX will then not close the cell of the outer table until the matching end group is seen, even if it sees a & or \cr token that would normally end the cell.

\def\tabc{%
{\ifnum0=`}\fi 
\leavevmode
\vtop\bgroup\let\\\cr
\halign\bgroup&\quad\hfil\ignorespaces##\unskip\hfil\quad\cr}

\def\endtabc{%
\crcr\egroup
\egroup
\ifnum0=`{\fi}}



\tabc
\tabc a & b \cr x & y \endtabc & 2 & 3\\
aaa & bbb & ccc
\endtabc



\bye

Answer 4

One of TeX's core features, delimited arguments/parameters, doesn't seem to be widely used in LaTeX user code, so some of the common idioms related to them may not be immediately clear when you encounter them in package code. The syntax and argument reading for those parameters is explained in this answer, so I'd like to concentrate on some examples of more or less common idioms using here.

Special argument delimiters

Special argument delimiters are usually implemented using delimited parameters, most notably optional parameters [...]. The typical pattern for LaTeX macros with optional arguments looks like

\def\foo{\@ifnextchar[{\foo@aux}{\foo@aux[default]}}
\def\foo@aux[#1]{Foo with parameter `#1'}

where \foo tests if the following character is [ and then passes over to \foo@aux which uses delimited parameters to read the following optional argument. Arguments with delimiters like (...) or <...> work similarly.

Skipping code until end marker

A common idiom is to use delimited parameters to skip parts of the code. As explained in Ryan Reich's answer, sometimes we have to get rid of unwanted tokens like an extra \else or \fi. The answer presents one approach to deal with them.

Another one is to define helper macros that use delimited parameters to skip through the conditional up to the final \fi, and then move the corresponding parameter after the \fi that is required to end the conditional. Using the macros

\def\handletrue#1\else#2\fi{\fi#1}
\def\handlefalse#1\fi{\fi#1}

we get the following expansions series for different conditions:

    \iftrue \handletrue{\textbf}\else \handlefalse{\textit}\fi{foo}
--> \handletrue{\textbf}\else \handlefalse{\textit}\fi{foo}
--> \fi\textbf{foo}
--> \textbf{foo}

    \iffalse \handletrue{\textbf}\else \handlefalse{\textit}\fi{bar}
--> \handlefalse{\textit}\fi{bar}
--> \fi\textit{bar}
--> \textit{bar}

thus correctly outputting "foo"/"bar". The lazylist package uses something similar:

\def\TeXif#1{#1\gobblefalse\else\gobbletrue\fi}
\def\gobblefalse\else\gobbletrue\fi#1#2{\fi#1}
\def\gobbletrue\fi#1#2{\fi#2}

A second use case of this idea are multiway branches (switch/case). TeX doesn't provide this control structure natively, so usually nested conditionals are used to avoid redundant tests in the code. Again, delimited parameters can be useful here. l3regex, for example, defines macros \__l3regex_break_true:w and \__l3regex_break_point:TF (renamed below for readability) which can be used to emulate a multiway branch:

\def\breaktrue#1\breakpoint#2#3{#2}
\def\breakpoint#1#2{#2}

\def\testchar{y}
\if\testchar x
    An `x` was found.\breaktrue
\fi
\if\testchar y
    An `y` was found.\breaktrue
\fi
\if\testchar z
    An `y` was found.\breaktrue
\fi
\breakpoint{\fi}{No character matched.}

Whenever an \breaktrue is found, it will skip anything up to the next \breakpoint, in particular all the remaining alternative tests. If any of the tests was successful, the first parameter of \breakpoint is inserted into the token stream (here the skipped \fi for the successful test must be reinserted), otherwise a common "else branch" is used.

Splitting token lists

Delimited parameters are also often used for testing if a certain separator token occurs in a list of tokens, such that the sublists before and after that token can be handled in a special way. As an example, say we want to replace all occurences of \textbf{...} in a token list by \textit{...}. We start with a helper macro \@replacebold that uses delimited parameters to split a token list, passed to it without surrounding braces:

\def\@replacebold#1\textbf#2#3\@end{...}

\textbf here delimits any tokens that occur before the first occurence of this control sequence in the list (#1), \@end is the delimiter for all the rest. As \textbf takes a single argument, we add another parameter #2 to match on this argument. Finally, #3 matches anything after the bold part. With this decomposition we can easily replace the bold text with an italic one and handle the rest of the list recursively.

Delimited parameters are mandatory in the actual argument if used in the macro definition, so we still need to handle the case in which \textbf{...} doesn't occur in the argument at all (which is also the case at the end of recursion). For this we already pass \@replacebold a single, faked call of \textbf which will match in any case:

\def\replacebold#1{\@replacebold#1\textbf{}\@end}

If this final occurrence is reached when calling \@replacebold, #3 will be empty as \textbf{} is immediately followed by \@end. We can use this fact for our test if another recursion step is necessary. Putting all pieces together, we might define \replacebold as follows:

\def\replacebold#1{\@replacebold#1\textbf{}\@end}
\def\@replacebold#1\textbf#2#3\@end{%
    #1%
    % Real code should use a saner test for emptiness here
    \ifx\relax#3\relax%
        \expandafter\@gobble
    \else
        \textit{#2}%
        \expandafter\@firstofone
    \fi
    {\@replacebold#3\@end}%
}

This idiom is used quite frequently in many variations. LaTeX's \@for loop macro, for example, uses a more sophisticated version to split a comma-separated list into its fields. You might also enjoy trying to find out how the various \quark_if_recursion functions form expl3 use this pattern for looping.

Other uses

Paragraphs as arguments: When TeX finds an end-of-line character in the input text that is immediately followed by another such character, it converts both into a \par token. This token can also be used as a delimited parameter. For example, the macro

\def\warning#1\par{%
    \textbf{Attention!} {\itshape #1\par}
}

could be used like \warning Do not use ... and will then read all text up to the end of the current paragraph, i.e. the next explict \par or empty line in the input file.

Simple loops: Plain TeX defines a simple loop macro \loop ... \repeat which uses a delimiter to mark the end of the loop body:

\def\loop#1\repeat{\def\body{#1}\iterate}
\def\iterate{\body \let\next\iterate \else\let\next\relax\fi \next}
\let\repeat=\fi % this makes \loop...\if...\repeat skippable

\count0=1
\loop
    ... use \count0 here ...
\ifnum\count0<10
    \advance \count0 by 1
\repeat

Use without parameter: Plain TeX also has a special use of delimiters in an internal macro to make sure that the command name passed to \newif actually starts with the letters i f. The macro \@if then gobbles those letters and builds a control sequence from the rest of the letters:

\def\@if#1#2{\csname\expandafter\if@\string#1#2\endcsname}
{\uccode`1=`i \uccode`2=`f \uppercase{\gdef\if@12{}}}
% The letters "12" here -----------------------^^
% are converted to the letters "if" with catcode 12 and
% act as delimiter (no actual argument is used).

Answer 5

I list the idioms that are used to solve a specific problem, and _{(at least in my opinion)} it's very hard to come up with the idiom when you need face the problem.

_{See also the linked questions in the comments.}

_{I might have missed a lot, because since now I know the idiom by definition I can solve the problem.}

What you want	How to solve it	Wrapped in LaTeX function
Pass result of unexpandable function as argument of other function	Make the first function store the result somewhere	N/A
Undef control sequence	\let \a = \undefined (where \undefined is undefined)	\cs_undefine:N (act globally)
Execute something with special category code (inject tokens with special category code into token list/code)	lowercase trick	regex_replace, etc.
Expand the second argument once / expandafter skip through lots of tokens	Argument swapping trick	\exp_args:Nno
(similar to above) Inject value of token list deep inside another token list	Define auxiliary function/ \edef \noexpand	none (occasionally can use regex. Note that \tl_replace_all does not recurse)
Compare two token list equal	Store them into two macro then \ifx	\tl_if_eq:nnTF
Check token list empty (expandably)	Basic idea: \if \relax <token list> \relax <true code> \else <false code> \fi (also detokenize, see linked post ← for details)	\tl_if_empty:n
Iterate over items of token list	Recursive macro, take items one by one (see "an example of recursion with quarks" section in interface3.pdf)	\tl_map_function:nN
Preserve braces around delimited arguments	Why does TeX remove braces around delimited arguments?	none
Iterate over tokens of token list	(see below)	(see below)
Create control sequence dynamically without possibly define it to \relax	wrap the expansion in begingroup...endgroup	none
Build token list progressively, allow partial to be unbalanced	Use \iffalse { \else } \fi trick then x-expand the complete token list. See also brace trick	gtl package (drawbacks: cannot convert back to normal tl, ignore charcode of {})
Convert given charcode (e.g. 97) to token	\lowercase trick	(no wrapper)
Convert given charcode (e.g. 97) to token expandably	\Ucharcat (XeTeX) / tex.cprint (LuaTeX) / consider 256 cases	\char_generate:nn

Regarding token list manipulation, there are generally 3 cases

What you want	How to solve it	Wrapped in LaTeX function
1. expandable, the token list is provided as argument		etl package, etc.
Check if first token in tl is explicit space	Use #1 delimited by explicit space then check #1 empty	\tl_if_head_is_space:n
Absorb the space	define macro delimited by explicit space	none
Check if first token in tl is brace group	\string the first token and check for imbalance	\tl_if_head_is_group:n
Get charcode of { or }	Get \string-ification of first opening brace in argument?/Get \string-ification of first opening brace's matching closing brace in argument?
2. unexpandable, the token list is provided as argument		\tl_analysis_map_inline, tokcycle package
get meaning of following token	\futurelet	peek_after:Nw (low-level)
get meaning of 2-next tokens	\afterassignment + \futurelet combination	N/A
distinguish tokens with same meaning, e.g. explicit token and implicit token as active character	temporarily redefine some suspects to undefined -- read source3.pdf on \tl_analysis_map_inline and \peek_analysis_map_inline for more details; then use \string	N/A
3. unexpandable, the token list is the following tokens in the input stream		peek_analysis_map_inline (does not work in all cases, read source3)
4. (extra) expandable, the token list is the following tokens in the input stream	directlua / immediateassignment (LuaTeX engine only)	none

And some things that you don't really need (there are clunky workarounds), but would be more convenient if you can:

What you want	How to solve it	Wrapped in LaTeX function
Compare two token list equal (expandably)	????	\etl_if_eq:nnTF (etl package)
Do complex working in one expansion step	\romannumeral trick / \expanded primitive	not really. (in expl3 you can just use :e-type argument or \exp:w ⟨expandable tokens whose expansion at some stage yields a leading token \exp_end: ⟩)
Modify value/state in expansion-only context	\csname (something) \endcsname defines (something) to be \relax if it was undefined; \ifcsname to check	l3flag package in expl3 (nonstandard trick, normally this is handled by passing argument values like functional programming)
Make macros expandable	See Tricks to make macros expandable	not really?

These things you may be able to figure out yourself once you know about the feature:

What you want	How to solve it	Wrapped in LaTeX function
Expand argument once before giving to function	\expandafter \function \expandafter { \argument }	\exp_args:No
Expand argument fully before giving to function	\edef or \expanded	\exp_args:Nx
Convert a to charcode (e.g. 97)	\number `a (remark: `a by itself is already an (integer denotation), like "FF for 255)	\int_eval:n