# Expansion of first token in a tabular cell

### Background

I currently do some work on the collcell package which collects the content of a tabular/array cell and provides it to a user defined macro, e.g. using >{\collectcell\mymacro}c<{\endcollectcell}.

### The Problem

I discovered that the very first token of every cell is expanded prior to the insertion of the >{...} code. I'm not sure how and where this is done. Does anyone know:

1. What expands this token (i.e. the array/tabular LaTeX code or the plainTeX \halign code or else?) and where
2. Why this is done (I have some wage idea, but I'm very interested about "the official" reason)
3. Can I stop it from being expanded somehow? I mean for my package in an user friendly way?

(I really have to read the relevant part of The TeXBook again when I have more time.)

-

It is done by TeX itself, and it will be hard to stop it. From the TeXbook, double dangerous bend on page 240:

once the \cr at the end of the preamble has been sensed, TeX must look ahead to see if the next token is \noalign or \omit, and macros are expanded until the next non-space token is found. If the token doesn’t turn out to be \noalign or \omit, it is put back to be read again, and TeX begins to read the template (still expanding macros).

The problem seems to be that the stuff inserted by >{...} is put into the preamble of an \halign, and the expansion happens before TeX reads the template from the preamble.

-
Thanks Hendrik, it turned out that you could teach me something about TeX after all. ;-) –  Martin Scharrer Feb 17 '11 at 17:50
@Hendrik: Can redefining \cr to \cr\relax work? –  Bruno Le Floch Feb 17 '11 at 18:58
@Bruno: Might be possible, but you'd also have to work on the &: what I wrote applies to every cell. –  Hendrik Vogt Feb 17 '11 at 19:03
@Bruno: I think that would break all \hline etc. after \\  because the \halign inside it isn't found any longer. –  Martin Scharrer Feb 17 '11 at 19:54
@Bruno, Hendrik: There is the additional issue of an expanded \\  if the last cell is empty, i.e. it is expanded when the \noalign after the last & is searched for. In this case my macro never sees the \\  and swallows hard on its content (\iffalse{\fi\ifnum0}\fi or so). I'm planning to \robustify\@arraycr (etoolbox) to fix this. Any objections? –  Martin Scharrer Feb 17 '11 at 19:59

TeX's tables are strange beasts with respect to expansion. Here is a plain TeX table with a right-aligned and a left-aligned columns

\halign{\hfil #&#\hfil \cr
a & bc \cr
de & f \cr}


The first line is the "preamble", which tells TeX what to do with the material in each cell, represented by #. So the first line becomes \hfil a&bc\hfil \cr. The \hfil stand for blank space that can stretch to fill an arbitrary distance, hence aligning the cell contents.

Taking this approach as given, let us try to invent how rules could be added to the language, with Knuth's hat on. [Admittedly, I'm likely to err in some of the resons that led Knuth and later engine developpers to make various decisions, but I think my picture is consistent, albeit full of anachronisms.]

Rules should be added between two lines, in other words, just after \cr. We postulate a hypothetical \hline, added after the \cr to produce a horizontal rule:

% Fake TeX code
\halign{\hfil #&#\hfil \cr
a & bc\cr
\hline
de & f\cr
\hline}


In the approach described so far, TeX would take \hline de as the cell's contents, and the second line would become \hfil \hline de&f\hfil\cr. In that particular case, we could still manage by making \hline insert material before the current line. But the trailing \hline is problematic: TeX would realize too late that there is no cell here: it would insert \hfil for the start of the first cell's preamble, then see \hline, insert the rule before the current line, see }, and have to back up somehow the \hfil.

A saner approach would be that when TeX sees \cr, it looks at what follows for an \hline. Should \hline be a primitive? That would only allow for the types of rules or inter-row material which have been hard-coded in the engine itself. No. A much better solution, which Knuth chose, is to allow for arbitrary material, with \noalign. Then there is no need to also provide \hline, which is simply a horizontal rule (\hrule) which is not aligned. So the real plain TeX way of having a line (well, two) is

\halign{#&#\cr
a & b \cr
\noalign{\hrule}
c & d \cr
\noalign{\hrule}}


Most will complain that the rules are too close to the text, but we are interested for now in expansion issues, not typography. Naturally, one may want to provide a shorthand for \noalign{\hrule}, say, \hline:

\def\hline{\noalign{\hrule}}
\halign{\hfil #&#\hfil\cr
a & b \cr \hline
c & d \cr \hline}


Again, we end up with the question of how TeX can know that this \hline macro hides a \noalign, and how TeX knows that it shouldn't just insert \hfil right away. The answer is that to find the \noalign, TeX expands macros after \cr, before inserting the material from the preamble. The same happens at each cell to look for \omit, but I won't delve on that.

This leads to problems: for instance, a macro which behaves differently in math mode than in text mode, should not start with \ifmmode but with \relax\ifmmode.

\def\foo{\relax\ifmmode x^2\else the square of $x$\fi}
\halign{\hfil $#$\hfil &\hfil $#$\hfil \cr
\foo & y^2 \cr
z-2 & t\cr}


TeX expands the first token after the \cr, which is \foo, then sees \relax. This stops the expansion, it is not \noalign nor \omit, so the preamble is inserted, entering math mode. The \ifmmode test is then performed. Without the \relax, \ifmmode would be evaluated before the preamble is inserted, and be false.

To help fight against this expansion, the eTeX programmers decided (in version 2, according to the eTeX manual) that \protected macros would stop the expansion in this situation. This is somewhat inconsistent with how eTeX expands in other full-expansion-from-the-left settings such as \romannumeral-q. In Martin's case (see his answer to the current question), this ends up being very useful since he can stop the expansion using a protected macro. In other cases (see some of Peter Grill's questions, for instance one about how to provide a wrapper for \cmidrule), we would like TeX to try harder to find the hidden \noalign.

-
Nice explanation. I've always wondered about the \relax in the definition of \strut on p. 353 which doesn't really explain it: "The \relax in this macro and in others below is necessary in case \strut appears first in an alignment entry, because TeX is in a somewhat unpredictable mode at such times". –  morbusg Jul 15 '12 at 16:23

Thanks to Hendrik's answer and the chat with him and Joseph Wright as well as the comment from Bruno Le Floch, I now solved all the issues related to the expansion in tabulars, i.e. in the underlying plainTeX primitive \halign. I like to list them here in case there are useful for others.

Just to summaries: The issue here was that \halign expands the tokens after \cr (which is more or less the plainTeX version of \\) and after & to see if a \noalign follows. This expansion stops when a non-expandable token is found. The collcell package wants to collect this tokens unexpanded. There is also the issue of the \\ getting expanded if the cell only is empty or contains only macros which expand to nothing or spaces, like \empty or \space. This is an issue because the code looks for \\ as an end-marker and didn't liked the included dirty tricks much.

The solutions I came up with now are:

• The \\ macro is \robustify-ed with the help of the etoolbox package. This way it isn't expanded by \halign but still works as normal otherwise.
• A special start-marker will be provided which is also \protected to be unexpandable. The collcell code will explicitly ignore (i.e. gobble) it when it is collecting the cell content. This allows users to place it in cells they want to be collected without any expansion and a \relax isn't suitable. This is necessary because special macros like \texttiming (tikz-timing package) must not be fed with unexpandable macros.
This isn't optimal, because it still requires some action by the user, but I can't see any other solution. Redefining & is not really an option IMHO.
• After the \\ is found by collcell the collection of tokens is stopped, the there included \cr is temporary redefined to re-start the collection after the original \cr is expanded (using \expandafter) which makes TeX insert the tokens defined by <{ }. This way these tokens are now also collected, which is important to support cascaded >{}/<{}.

“… and after & to see if a \halign follows…” I think you mean \noalign follows, since there can't be a \halign there? –  morbusg Feb 26 '11 at 21:46
@morbusg: Thanks, I fixed that. I was supposed to be \noalign. –  Martin Scharrer Feb 26 '11 at 21:48