Why six \noexpand in a row?

Question

Trying to understand TeX's rules for parsing dimensions, I came up with the following weird behaviour:

\dimen0=1\noexpand\noexpand\noexpand\noexpand\noexpand\empty pt

is valid, but

\dimen0=1\noexpand\noexpand\noexpand\noexpand\noexpand\noexpand\empty pt

yields

! Illegal unit of measure (pt inserted).
<to be read again> 
                   p
<to be read again> 
                   t
<*> ...oexpand\noexpand\noexpand\noexpand\empty pt

? 

What causes the difference?

Answer 1

This was tricky to track down. Notice that we get an error after different numbers of \noexpand depending on the unit, which means that TeX does a different number of expansions depending on the unit. To experiment with this, let's set the active character ~ to be \noexpand, then try various assignments, increasing the number of ~ until TeX complains. Here are the maximum numbers of ~ for the original TeX engine:

\let~\noexpand
\dimen0=1~~\dimen1
\dimen0=1~~~em
\dimen0=1~~~~ex
\dimen0=1~~~~~true~pt % (and others)
\dimen0=1~~~~~~pt
\dimen0=1~~~~~~~in
\dimen0=1~~~~~~~~pc
\dimen0=1~~~~~~~~~cm
\dimen0=1~~~~~~~~~~mm
\dimen0=1~~~~~~~~~~~bp
\dimen0=1~~~~~~~~~~~~dd
\dimen0=1~~~~~~~~~~~~~cc
\dimen0=1~~~~~~~~~~~~~~sp
\dimen0=1~~~~~~~~~~~~~~~ % crashes after reading the last ~.

Clearly, all units are not created equal. The key is found in the tex.web file: the code TeX uses for scanning a unit is copied below. Let's comment on the case of

\dimen0=1~~~~~~~pt

for instance, which leads to an error (with 7 ~). TeX reads the number 1, expands the first ~ which temporarily turns the second ~ to \relax, stopping the expansion of 1: the number is complete. The:

@<Scan units and set |cur_val| to $x\cdot(|cur_val|+f/2^{16})$...@>=

This is essentially a "title".

if inf then @<Scan for \(f)\.{fil} units; |goto attach_fraction| if found@>;

Dimension assignments do not allow infinite glue, so this part is skipped.

@<Scan for \(u)units that are internal dimensions;
  |goto attach_sign| with |cur_val| set if found@>;

This scans for a unit such as \dimen..., expanding our second ~ which turns the next ~ to \relax (note that in the first line of our example file the ~ acts on \dimen and does nothing, so TeX sees \dimen). Then TeX looks for em, then for ex, "killing" two more ~. We are left with 3.

if mu then @<Scan for \(m)\.{mu} units and |goto attach_fraction|@>;

Skipped: this is not a muskip assignment

if scan_keyword("true") then @<Adjust \(f)for the magnification ratio@>;
@.true@>

TeX tries to scan true and fails again, leaving us with 2 ~.

if scan_keyword("pt") then goto attach_fraction; {the easy case}
@.pt@>

TeX looks for pt, expands, but the next ~ turns the last ~ to \relax, hiding the pt which follows. Yet another failure.

@<Scan for \(a)all other units and adjust |cur_val| and |f| accordingly;
  |goto done| in the case of scaled points@>;

The code called there tries various units one by one: in, pc, cm, mm, bp, dd, cc, sp. Our last ~ does nothing to the following p, and TeX keeps seeing a p, trying to match it with the units. When scanning for pc, TeX recognizes the p, but is then disappointed by the t. Eventually, TeX gives up. Note that the error message in this case shows both p and t as "to be read again". If you add two more ~, TeX hasn't reached the p yet when trying to parse pc, so only the p will be "to be read again".

attach_fraction: if cur_val>=@'40000 then arith_error:=true
else cur_val:=cur_val*unity+f;
done:

Rest of the code, we don't care here.

As a side note: since eTeX scans a few additional units than TeX, the following crashes at 14 ~ with TeX but 18 with eTeX:

\let~\noexpand
\dimen0=1~~~~~~~~~~~~~~~~~sp