Strange behaviour with PGF/TikZ and Plain TeX output routine

Question

I noticed a strange behavior when running PGF/tikZ with pdfTeX on my Debian box : a simple session without tikz:

pdftex
**\relax
*Hello World
*\bye

do output the page number on the page footer. On the other hand, loading tikz:

pdftex
**\relax
*\input tikz.tex
*Hello World
*\bye`

does make the page number on the footer disappear. The page size does not seems to be the cause.

Does any of you have any idea of the cause behind this behaviour?

Answer 1

The problem is what pgf does with \shipout. It redefines it to

\afterassignment\pgfutil@@EveryShipout@Test\setbox 255=

That would be fine if the content of box 255 was set up when plain calls \shipout. However, it's not:

\plainoutput ->\shipout \vbox {\makeheadline \pagebody \makefootline }\advancepageno    
  \ifnum \outputpenalty >-\@MM \else \dosupereject \fi

Notice the \makefootline here in particular. What goes wrong is that \afterassignment inserts the token \pgfutil@@EveryShipout@Test not after the closing } but after the opening { bracket, i.e. just before \makeheadline. (This behaviour is documented in the usual places, for example TeX by Topic.)

This is all an issue because pgf then does some tests on box 255 before firing the \shipout primitive as part of the expansion of \pgfutil@@EveryShipout@Test. Thus the shipout is taking place before addition of the head/footlines: bad news. (You can see this in \tracingall output, where the box appears in completely the wrong place.)

Answer 2

You can detect that the tokens inserted by \afterasignment are in a new vbox list rather than the current list and if so, delay them with \aftergroup. Two versions, one for classic TeX then after \bye a version for etex.

\input tikz.tex
\catcode`\@=11
\def\shipout{\vskip1sp \afterassignment \zzz \setbox 255= }

\def\zzz{%
\ifdim\lastskip=\z@\expandafter\aftergroup\fi
\zzzz}

\def\zzzz{%
\unskip
\pgfutil@@EveryShipout@Test}

Hello World
\bye


\input tikz.tex
\catcode`\@=11
\def\shipout{\afterassignment \zzz \setbox 255= }

\def\zzz{%
\ifnum\currentgrouptype=4 \expandafter\aftergroup\fi
\pgfutil@@EveryShipout@Test }

Hello World
\bye

Answer 3

The problem is, as Joseph Wright detected (beating me for one minute), that PGF makes some wrong assumptions about how the \shipout operation is called. If it was, like in LaTeX,

\shipout\box255

(or some other box register, it doesn't matter), the redefinition of \shipout would work. But the redefinition is

\def\shipout{\afterassignment\pgfutil@@EveryShipout@Test\setbox255= }

which goes wrong when \shipout\vbox{\makeheadline\pagecontents\makefootline} is called by the output routine of Plain TeX. Indeed, the TeXbook says

\afterassignment<token>. The <token> is saved in a special place; it will be inserted back into the input just after the next assignment command has been performed. An assignment need not follow immediately; if another \afterassignment is performed before the next assignment, the second one overrides the first. If the next assignment is a \setbox, and if the assigned <box> is \hbox or \vbox or \vtop, the <token> will be inserted just after the { in the box construction, not after the }; it will also come just before any tokens inserted by \everyhbox or \everyvbox.

Thus the test is performed at the wrong time, before \makeheadline and \makefootline are executed, so that headers and footers will not appear.

PGF makes also another wrong assumptions, that one always calls \shipout\box<register> where the box is a \vbox. Thus a simple

\setbox0=\hbox{A}
\shipout\box0

would fail (Incompatible list can't be unboxed). Here's a possible fix (using ideas from David Carlisle's answer).

\input pgf

\catcode`@=11
\newbox\pgfutil@@Output@Box
\def\shipout{%
  \ifhmode\hskip\else\vskip\fi 1sp 
  \afterassignment\pgfutil@@Delayed@EveryShipout@Test
  \setbox\pgfutil@@Output@Box=}
\def\pgfutil@@Delayed@EveryShipout@Test{%
  \ifdim\lastskip=\z@ % it was \shipout\vbox or \shipout\hbox
    \expandafter\aftergroup
  \fi
  \pgfutil@@EveryShipout@Test}
\def\pgfutil@@EveryShipout@Test{%
  \unskip % remove the skip used as signal
  \ifvoid\pgfutil@@Output@Box
    \expandafter\aftergroup
  \fi
  \pgfutil@EveryShipout@Output}
\def\pgfutil@EveryShipout@Output{%
  \setbox\pgfutil@@Output@Box=\vbox{
    \setbox\z@=\hbox{%
      \pgfutil@abe
      \unhbox\pgfutil@abb
      \pgfutil@abc
      \global\let\pgfutil@abc\pgfutil@empty
    }%
    \wd\z@=\z@\ht\z@=\z@\dp\z@=\z@\box\z@
    \ifhbox\pgfutil@@Output@Box\unhbox\else\unvbox\fi\pgfutil@@Output@Box
  }%
  \pgfutil@@EveryShipout@Org@Shipout\box\pgfutil@@Output@Box
}
\catcode`@=12

%\def\bxxx{\box255}\output={\shipout\bxxx}
%\output={\shipout\relax\box255}

\setbox0=\hbox{A}
\shipout\box0

Hello world
\bye

This would probably give wrong results in case of \shipout\vtop{...}, which however seems not to be something which is done frequently. A test whether we are in a \vtop group, with e-TeX's \ifnum\currentgrouptype=5 might be made in the \ifdim\lastskip=\z@ conditional, adjusting things later when

\setbox\pgfutil@@Output@Box=\vbox{

is executed.

The "naked" shipout operation is performed correctly. Also changing the output routine shows that this might work in all normal situations.

A better way could be using the atbegshi package by Heiko Oberdiek, without reinventing the wheel.