Where do I find \futurelet's nasty behaviour documented?

Question

Until recently, I thought that execution of

\futurelet\testtoken<token1><token2>...

has the following effect: The control sequence \testtoken is \let to <token2> and \futurelet\testtoken is stripped from the input stream, so TeX continues with <token1><token2>....

In this answer of mine I observed (with TH's help) that this is not quite true with respect to catcodes, and in this nice answer Philippe explains that things can also go terribly wrong in nested alignments. For the sake of completeness let me give two short examples here.

\def\activateA{\catcode`A=\active}
{\activateA
 \gdefA{undocumented behaviour?}
}
An \futurelet\testtoken\activateA A
\bye

The expected output is "An undocumented behaviour?", but this one gets only when \futurelet\testtoken is removed. With the \futurelet, the output is just "An A". The reason seems to be that \futurelet fixes the catcode of the last A. (One still gets that \testtoken=the letter A.)

The second example is a bit longer and more involved.

\def\begintestalign{\show\testtoken
    $\vcenter\bgroup\halign\bgroup##&##\cr}
\def\endtestalign{\egroup\egroup$}
\halign{#&#\cr
a & \futurelet\testtoken
    \begintestalign
         & c  \cr
    test & de \cr
    \endtestalign \cr
}
\bye

The expected output is (and one would expect \testtoken=alignment tab character &), but instead one gets \testtoken=\outer endtemplate, and then

! Emergency stop.
<recently read> \endtemplate 

<template> \endtemplate 

l.7          &
               c  \cr
!  ==> Fatal error occurred, no output PDF file produced!

I can't fully understand what's happening here, but my main question is: Where do I find this behaviour of \futurelet documented?

(And honestly, I'd really like to have a \futurelet that behaves as expected; the implemented behaviour is rather mean, and I wonder if that's a bug or a feature.)

Answer 1

There are actually two parts to your question that are interacting a bit strangely.

For the first example, the behavior is completely sensible. As others have observed, the A is tokenized and given category code 11. Once a character token is tokenized, it never loses its category code (except for e-TeX's \detokenize extension).

The second example is much trickier. My initial reaction was that it should work as you want. Unfortunately, TeX treats \halign in a very strange way. Let me quote from The TeXbook, page 248.

You have to be careful with the use of & and \span and \cr because these tokens are intercepted by TeX's scanner even when it is not expanding macros. For example, if you say ‘\let\x=\span’ in the midst of an alignment entry, TeX will think that the ‘\span’ ends the entry so \x will become equal to the first token following the ‘#’ in the template. You can hide this \span by putting it in braces; e.g., ‘{\global\let\x=\span}’. (And Appendix D explains how to avoid \global here.)

This is very surprising, but it explains the behavior you see. Namely, when TeX encounters the & when looking for the token to \let to \testtoken, it sees a & and so inserts the rest of the template (empty in this case) followed by \endtemplate—the command that causes the contents of the alignment entry to be typeset in an "unset" box (meaning the glue hasn't been set yet).

Answer 2

IMHO the shown behavior in the first example is understandable, i.e. "correct". The TeXbook says about the \futurelet macro (p.207):

TEX also allows the construction \futurelet\cs<token1><token2>, which has the effect of \let\cs = <token2><token1><token2>.

So we are talking about tokens, which already have their catcode assigned. In your example A is the <token2> and must be read by TeX in order to assign it to the command sequence \testtoken. The macro \activateA is not executed yet so the catcode of A is still letter. When it is placed back into the input stream (which is the main point here) the catcode change in \activeA doesn't effect it any longer.

So actually \futurelet\cs<token1><token2> does not have the 100% same effect of \let\cs = <token2><token1><token2> because in the second case <token1> can still modify the catcodes in effect for reading the second <token2>.

You second example is much more tricky. IMHO you are falling victim to automatically inserted code after (and before) the cell content. Therefore when \futurelet is executed there is really an \outer endtemplate behind it. This is then a behaviour of \halign and shouldn't be blamed on \futurelet. As you know my knowledge about \halign is still limited so I can't example the resulting error as well.

I would say if there is a documentation of this behaviour, it is in The TeXbook. I don't think eTeX changed the behavior of \futurelet in any way.

Answer 3

Disclaimer: I have no references to support my claims below.

Let me start with a simple example:

\def\showtoken{\show\testtoken}
\halign{#b\cr
  a\futurelet\testtoken\showtoken\cr}

When it reads the first cell, TeX sees the unexpandable token a, so it knows that there is no \omit. It then inserts the part of the template before #, namely, nothing, and it prepares to insert the part of the template after #, namely, b, when the cell finishes.

It then reads the \futurelet, and executes it. Thus TeX looks for three tokens. The first and second are \testtoken and \showtoken. The third is not \cr: when it sees \cr in this context, TeX inserts what it desperately wants to insert: the end of the template, b. So we have \futurelet \testtoken \showtoken b. Then \testtoken is \let to b, and we are shown b by the macro \showtoken.

An interesting combination of \futurelet and \afterassignment allows you to peek two tokens ahead, and we check that \futurelet indeed reads the tokens \testtoken, b and c and assigns \testtoken=c.

\def\showtoken{\show\testtoken}
\halign{#bcd\cr
  a\afterassignment\showtoken\futurelet\testtoken\cr}

Now delete cd in the template. TeX should print on your terminal

> \testtoken=\outer endtemplate:
.
\showtoken ->\show \testtoken 

<to be read again> 
                   b
<to be read again> 
                   \endtemplate 
<template> b\endtemplate 

l.3 ...assignment\showtoken\futurelet\testtoken\cr
                                                  }

As you can see, the tokens inserted by TeX at the end of the cell are ended by \endtemplate, which is some internal token. Removing even the b, we see that \futurelet reads \testtoken, the internal \endtemplate, and the following token, }. Then TeX crashes.

A variant on this is to grab the \endtemplate using \let, and redo it to close the template. Playing around with the code below, I notice that if we put anything else than \testtoken (which holds the \endtemplate) in the last position of \showdotoken, TeX crashes. I have no idea why.

\def\showdotoken{\show\testtoken \testtoken}
\halign{#\cr
  a\afterassignment\showdotoken\let\testtoken\cr
}

It seems that TeX crashes whenever we try to reach out after the end of a cell in an alignment: both \halign below crash.

\halign{#\cr
  a\futurelet\testtoken\cr}
\halign{#&#\cr
  a\futurelet\testtoken & text\cr}

Back on Hendrik's example.

\def\begintestalign{\show\testtoken
    $\vcenter\bgroup\halign\bgroup##&##\cr}
\def\endtestalign{\egroup\egroup$}
\halign{#&#\cr
a & \futurelet\testtoken
    \begintestalign
         & c  \cr
    test & de \cr
    \endtestalign \cr
}

The \futurelet takes three tokens: \testtoken, \begintestalign, and the material that TeX is waiting to insert as soon as it sees either & or \cr (or \crcr), namely, nothing, followed by the internal \endtemplate. So that & is already converted to \endtemplate (and perhaps some \begin-next-template) of the first \halign. Then \begintestalign is read, creates a new inner \halign, and that \halign awaits a fresh &... it sees an old &, already belonging to the enclosing \halign.

All this might be related to an error mentioned in the TeXbook, p. 299:

Interwoven alignment preambles are not allowed.

If you have been so devious as to get this message, you will understand it, and you will deserve no sympathy.

Answer 4

futurelet deserved its own chapter in the TeXbook but perhaps Knuth underestimated its importance. Best article I came across so far is by Victor Eijkhout in the TUGboat Macros with optional arguments

Answer 5

I'm posting this fix as a separate answer, and as community wiki, in the hope that people can improve on it.

Since the second of Hendrik's problems happens when \futurelet reads & and triggers the end of the outside alignment cell, the idea is to make & an active character. Most of the time, we will \let it to an implicit alignment tab \ampImplicit. Just before using \futurelet, we change the definition of & to be something harmless, and we restore the usual definition after the \futurelet assignment.

The code below typesets what Hendrik expects, but the \testtoken is \relax, i.e. the safe control sequence that we defined & to be just before \futurelet. We could replace \let&\relax by \def&{something recognizable} if we care enough.

\let\ampImplicit&
\catcode`\&=13\relax
\let&\ampImplicit

\def\makeampImplicit{\iffalse{\fi \let&\ampImplicit \iffalse}\fi }
\def\safefuturelet{\relax
  \iffalse{\fi \let&\relax \iffalse}\fi 
  \afterassignment\makeampImplicit
  \futurelet
}

% Hendrik's code, with `\futurelet` replaced by `\safefuturelet`.

\def\begintestalign{\show\testtoken
    $\vcenter\bgroup\halign\bgroup##&##\cr}
\def\endtestalign{\egroup\egroup$}
\halign{#&#\cr
a & \safefuturelet\testtoken
    \begintestalign
         & c  \cr
    test & de \cr
    \endtestalign \cr
}

\bye

I have tested this idea of making & active when using amsmath, which uses \halign internally. We can of course change the catcode after loading the package, but I wanted to see what breaks if it is active and let to its non-active self from the start. So far, I found that amsmath would break because of a \csname ... & ... \endcsname construction: that works if & is an explicit alignment tab, but not when it is an active character let to an alignment tab. This can be fixed by defining &13 to be a macro expanding to &4 just before loading the package, then reverting to the "implicit character" definition so that alignment preambles work as expected.

\documentclass{article}

\def\ampMacro{&}
\let\ampImplicit&
\catcode`\&=13\relax

\let &\ampMacro
\usepackage{amsmath}
\let&\ampImplicit

\begin{document}
\begin{align}
  x^2+y^2 &= z^2 \\
  \intertext{but in general,}
  (a-b)^2 &\neq a^2 - b^2 
\end{align}
\end{document}