TeX's expansion rules: a case study with a token-list register

Question

I'm trying to figure out TeX's rules of expansion. Consider the following plain TeX manuscript:

\toks1{hello}%
\toks10{world}%
\def\mac{0}%
\the\toks1\noexpand\mac%
\bye

When this manuscript is processed with pdftex, the following pdf results:

and the log file contains the following warning:

Underfull \hbox (badness 10000) has occurred while \output is active
[]\tenrm world

\hbox(8.5+0.0)x469.75499
.\vbox(8.5+0.0)x0.0
.\tenrm w
.\kern-0.27779
.\tenrm o
.\tenrm r
.etc.

[1{/usr/local/texlive/2017/texmf-var/fonts/map/pdftex/updmap/pdftex.map}] )<./c
mr10.pfb>

If I replace both occurrences of 0 in the manuscript by 2, so that the manuscript becomes

\toks1{hello}%
\toks12{world}%
\def\mac{2}%
\the\toks1\noexpand\mac%
\bye

the pdf contains a single line:

hello2

and no warnings in the log file.

Could someone please explain to me why the pdf document looks the way it does, as well as what the warning means, and why it is being reported. Also, please walk me through the steps TeX goes through in processing my manuscript. Thanks.

Answer 1

I'll skip over the three assignments and go straight to

\the\toks1\noexpand\mac

The \the primitive inserts the content of a register: here a \toks. After the \toks we need a number, so TeX goes looking for one: it finds a literal 1 then a \noexpand, which temporarily turns \mac into \relax and so terminates the search for a number. \toks1 contains hello so that gets inserted (and starts a paragraph), then TeX re-reads \mac which is now 'back to normal': a macro expanding to 0. So the paragraph you see is hello0. (There is of course a paragraph indent as one would expect at the start of a paragraph.)

So why does world get inserted? That happens as you've picked \toks10: alter the number and it vanishes. In plain TeX, \headline is \toks10 so gets inserted as part of the page finalisation: it's got nothing to do with the specific input you have (other than that you've set it!). Of course, this is why we almost always use the allocation system for registers.

Here, \tracingall is probably your friend: the example is short and the trace is clear.