TeX macro idioms, or: understanding advanced macros

Question

I often find that when I look at solutions from the experts here, the macro definitions are completely inscrutable to me. Trying to use standard techniques one would use to understand code in most programming languages often isn't helpful; for example, I find that chasing definitions using show can bottom out in something that still doesn't make sense (to me).

Now, @David Carlisle has just very kindly taught me a TeX idiom which suddenly makes a lot of macros make sense. This question is largely motivated by a desire to share that with other people at my level. It's asking for descriptions of any common idioms that:

1. occur frequently in packages and answers from experts here
2. are doing something that isn't obvious from just understanding the commands. (For example, lccode used to something that has nothing to do with lowercase characters.)

The idiom I've just learnt is going into an answer below, and should help illustrate what I mean.

Answer 1

I suppose we should have mention of

{\ifnum0=`}\fi 

\ifnum0=`{\fi}

The brace group constructs that are endemic in packages related to tabular and related alignment constructs.

To understand these constructs it is best to start in the middle: the backtick construct `{ returns the character code of { which is 123 and in particular it is not 0 so

\ifnum0=`}\fi 

is like \iffalse\fi and expands to nothing. Similarly

\ifnum0=`{\fi

Thus the first construct expands to an explicit { and the second construct expands to an explicit }

there are simpler constructs that expand to explicit braces notably

{\iffalse}\fi

\iffalse{\fi}

However as we shall see they do not work in the tabular constructs the way that is needed.

Now consider a simple halign construct (just using plain TeX for simplicity) that defines centred columns and (like LaTeX) wants to use a local definition of \\ to end the row.

\def\tabc{%
\leavevmode
\vtop\bgroup\let\\\cr
\halign\bgroup&\quad\hfil\ignorespaces##\unskip\hfil\quad\cr}

\def\endtabc{%
\crcr\egroup
\egroup}



\tabc
1 & 2 & 3\\
aaa & bbb & ccc
\endtabc



\bye

This apparently works fine and produces

Then someone decides that they want to nest tables …

Replacing the 1 in the first cell by a nested 2x2 table:

\tabc
\tabc a & b \cr x & y \endtabc & 2 & 3\\
aaa & bbb & ccc
\endtabc

This fails with the error message

! Missing } inserted.
<inserted text> 
                }
<to be read again> 
                   \endtemplate 
<template> \unskip \hfil \quad \endtemplate 

\tabc ->\leavevmode \vtop \bgroup \let \\
                                         \cr \halign \bgroup &\quad \hfil \i...
l.14 \tabc
           a & b \cr x & y \endtabc & 2 & 3\\

The hint to understanding the error message is the \\ at the end of the line has cause \endtemplate to be read. What has happened is that the intended meaning of locally defining \\ has not happened: as soon as TeX saw the \\ (which is \let to \cr already because of the outer table, the cell of the outer table ended, the the \let then defined \unskip to be \hfil, the quad made a space, but then the cell ended in while the group started by \halign\bgroup is still open, so an error was generated.

One work-around is to nest the inner table in explicit braces:

\tabc
{\tabc a & b \cr x & y \endtabc} & 2 & 3\\
aaa & bbb & ccc
\endtabc

works as intended and produces

Sometimes you see table macros with similar requirements that nested instances be protected by explicit braces. But normally it is better to make the macros safe for nested use.

Due to the way that macro definitions are parsed, it is not possible to simply add a { to the definition of \tabc. Also the usual implicit brace \lbrace does not work here, note that the redefinition of \\ is already inside a group started by \lbrace but TeX's table cell scanner does not see that in the right way. It turns out that the construct at the start is exactly what is required. If we define the table macro as below then the nested table works as expected without the need for extra braces in the document: Once the inner \tabc is expanded in the first cell of the outer table the {\ifnum0= `}\fi is expanded and TeX will then not close the cell of the outer table until the matching end group is seen, even if it sees a & or \cr token that would normally end the cell.

\def\tabc{%
{\ifnum0=`}\fi 
\leavevmode
\vtop\bgroup\let\\\cr
\halign\bgroup&\quad\hfil\ignorespaces##\unskip\hfil\quad\cr}

\def\endtabc{%
\crcr\egroup
\egroup
\ifnum0=`{\fi}}



\tabc
\tabc a & b \cr x & y \endtabc & 2 & 3\\
aaa & bbb & ccc
\endtabc



\bye

Answer 2

One of the more frustrating features of TeX's macro-expansion based language is that the control code and the code it controls can interact unexpectedly. This means that if you want a conditional block \ifwhatever...\fi to control some macro \macro that affects stuff after the block, you need to leapfrog \macro to finish the conditional first:

\ifwhatever
 \expandafter\macro
\fi
<stuff taken as an argument by \macro>

You see, the \expandafter eats the \fi before \macro can. Using this trick, you can write some code that conditionally executes some following code. Say you have a conditional \ifwhatever, and you want to execute <code> if it is true, but not if it is false. Then you can write (here I am assuming \makeatletter is active):

\ifwhatever
 \expandafter\@firstofone
\else
 \expandafter\@gobble
\fi
{<code>}

This can be generalized if you want to select among two different pieces of code, which by the way is how the LaTeX-style conditionals \ifwhatever{<true code>}{<false code>} work:

\ifwhatever
 \expandafter\@firstoftwo
\else
 \expandafter\@secondoftwo
\fi
{<code 1>}
{<code 2>}

I wouldn't advise doing this with more than one conditional, though: if you want to execute <code> when \ifwhatever or \ifnot is true, you have to write:

\ifwhatever
 \expandafter\@gobble
\else
 \ifnot
   \expandafter\expandafter\expandafter\@firstoftwo
 \else
   \expandafter\expandafter\expandafter\@gobble
 \fi
\fi
{<code>}

This profusion of \expandafters is to skip \@firstoftwo or \@gobble and expand first one token, then another after it, before going back to expand the macro you actually want once there is nothing in its way.

Of course, the technique of using multiple \expandafters to skip ahead is itself an idiom. For example, you can expand the first token in a group once:

\expandafter{\macro}

or twice:

\expandafter\expandafter
\expandafter{\macro}

or three times:

\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter
\expandafter{\macro}

and you see the pattern. I may as well explain why this works. When the first \expandafter goes off, it expands in turn the third, then the first in the next row, then the first in the following row, and finally, \macro. The intervening tokens are not expanded in this "run", but just stuck back in order. The effect is that every other column of \expandafters vanishes, \macro is expanded once, and then TeX starts expanding again with what is now the first \expandater, which was the second. The configuration has become the one for expanding \macro twice, and so we end up expanding it three times.

Answer 3

lccode + lowercase trick

@David Carlisle's trick is used when you want to redefine the behaviour of a standard (say) ASCII character, such as - or \n. This is useful in e.g. verbatim-like environments, where you might want a newline to be processed in a non-standard way -- and I'm going to stick with that example for the rest of this post.

TeX lets you make any such character into a macro using \catcode and \active, which gives you a easy way of changing the behaviour of \n. The problem is that as soon as you've done that, then every newline in your source code will start behaving differently, which makes it rather hard to actually write anything. So you need some way of being able to use 'inactive' newlines in your source while treating any newlines typed by your user as 'active'.

As I understand it, the trick works as follows:

1. You pick an arbitrary active character which you don't need to use in your macro. ~ is a good choice because it is always active. If you need more, you make whatever characters you like active.

2. You "tell" TeX that the lowercase version of (say) ~ is \n, like this:

   \lccode`~=`\^^M 
   

   This is of course nothing to do with actual lowercase text.

3. You wrap your entire macro in \lowercase{...}. This means that whenever you use ~, TeX converts it into \n. And because ~ is active, it converts it into an active \n. This means, in particular, that you can redefine the behaviour of \n like this:

   \lowercase{
   ...
   \def~{...}
   ...
   }
   

   At the same time, normal newlines in your macro (like the one before the final }) remain inactive, and so are processed by TeX in the normal fashion.

[At least, that's it as I best understand it... ]