What does the new luatex primitive \deferred do?

Question

I recently noticed that a new primitive, \deferred, was added to luatex (relevant texlive git changes). The luatex manual is austere in its examples and explanations for most things, and I see essentially nothing about \deferred other than a mention as a prefix on page 47 (not sure if this is the same thing) and its appearance in the big list of primitives on page 207.

What does \deferred do? My naïve guess based on the git changes is that it's some counterpart to \immediate, possibly related to the new "late special" feature. Beyond what it does, does anyone have an example document in which the new primitive is used to do something not previously possible?

P.S. If your texlive is not new enough, you can see that it is actually a proper "command" with this example on texlive.net.

Answer 1

Exactly as muzimuzhi said: \deferred is a LuaTeX alternative to the shipout keyword. That is, both should do the exact same thing.

The shipout keyword was implemented in pdfTeX (and copy-pasted ported to XeTeX, e(u)-pTeX, and Aleph) because we needed a “late expansion” for \special. When Hans implemented it for LuaTeX, he probably thought a \deferred prefix is better because it's symmetric to \immediate. muzimuzhi already answered your question spot-on, so here's an explanation of what \deferred or the shipout keyword are for:

When you use \special, it fully expands the argument (pretty much like \expanded), then adds the expanded token list as a whatsit node to the output list. That node is then written as-is to the output .pdf or whatever format the engine produces.

The problem with that, is that when TeX is reading your \special, it doesn't know it what page that \special will be placed, so using things like \thepage will sometimes give the wrong information.

In the example (pdfTeX) document below, we are typesetting one long paragraph across three pages: at the time TeX reads the \special (when it is building the horizontal list for the paragraph), it is still processing the first paragraph in page 1 (first it builds the list of nodes, then it breaks the list into lines, then those lines across pages), so \count0 (the page number) is 1, so it colours the text red.

If you switch to \special shipout it will hold the token list unexpanded until the page shipout (when everything is in its final position in the page), and only then it expands the token list, and evaluates the value of \count0. At this point, TeX already knows it's on page , so it colours blue.

\input color.tex

\hsize=5cm \pdfpagewidth=10cm
\vsize=5cm \pdfpageheight=10cm

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\special{pdf:%
% \special shipout{pdf:%
  \ifnum\count0=1 1 \else 0 \fi 0 \ifnum\count0=1 0 \else 1 \fi rg
  \ifnum\count0=1 1 \else 0 \fi 0 \ifnum\count0=1 0 \else 1 \fi RG}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

\bye

With \special:

With \special shipout (or \deferred\special in LuaTeX):

All of this was already possible without these extensions, but it required you to run TeX multiple times to make TeX write your \special on the page, then find out where it ended up, and then use that information (stored in a temporary file) to do the right choice later. This extension lets you do all in one go, without extra data structures and build steps.

Answer 2

Combine the TeX Live Guide---2023, sec. 9.2 Present: 2023

Cross-engine extensions (except in original TeX and e-TeX): \special followed by a new keyword “shipout” delays expansion of the argument tokens until \shipout time, as with a non-\immediate\write.

and the LuaTeX Reference Manual (Feb 2023, v1.16), sec. 3.2.1 Less primitives

[...] Here we show the options that can be passed to the extensions. The shipout option is a compatibility feature. Instead one can use the deferred prefix.

\pdfextension literal
  [shipout] [ direct | page | raw ] { tokens }
...

I think it means, for example

• \pdfextension literal shipout direct {<tokens>} (using the new option shipout) and
• \deferred\pdfextenion literal direct {<tokens>} (using the new primitive \deferred acting as a prefix)

do the same thing.

Thus perhaps in LuaTeX Reference Manual it's better to use \deferred instead of the current deferred.