Section 23.7 of source3.pdf lists all possible tokens, such as the usual characters and control sequences, but also some of TeX's internal ones.

  1. Expanding \the\font results in a token that looks identical to the command that was used to select the current font (such as \tenrm) but it differs from it in shape.
  2. A “frozen” \relax, which differs from the primitive in shape (but has the same meaning), is inserted when the closing \fi of a conditional is encountered before the conditional is evaluated.
  3. Expanding \noexpand <token> (when the <token> is expandable) results in an internal token, displayed (temporarily) as \notexpanded: <token>, whose shape coincides with the <token> and whose meaning differs from \relax.
  4. An \outer endtemplate: can be encountered when peeking ahead at the next token; this expands to another internal token, end of alignment template.
  5. Tricky programming might access a frozen \endwrite.
  6. Some frozen tokens can only be accessed in interactive sessions: \cr, \right, \endgroup, \fi, \inaccessible.

How can I produce these tokens, either by \letting them to another control sequence, a macro containing one, or simply "seeing" them in some form?

3 Answers 3


Update: this thing is patched (by making \noexpand no-operation on \endwrite) in new versions of (PDF)TeX, but another segmentation fault is possible. Detail in https://topanswers.xyz/tex?q=5286#a5394.

To be fair, maybe Bruno Le Floch (member of the LaTeX3 team, probably also the one wrote that paragraph in interface3.pdf) discovered this one before me.

error message - Shortest code that raises a SIGSEGV - Code Golf Stack Exchange

To be precise, correcting a comment below: both \show ⟨\endwrite⟩ and \meaning ⟨\endwrite⟩ will not segmentation fault in Knuth TeX -- the former shows

> \endwrite=\outer macro:

but it does in PDFTeX. Only \let ⟨some assignable thing⟩ ⟨\endwrite⟩ (or executing ⟨\endwrite⟩ directly) segmentation fault Knuth TeX.

No idea why nobody have posted an answer for the \endwrite token, but here it is.

%! TEX program = pdflatex

% starting state of the input stream:
%     \weird } \endwrite ...

\def \weird {   \expandafter \weirda \expandafter { \iffalse } \fi }

%   → \weirda { } \endwrite ...

\def \weirda #1  {  \expandafter \weirdb  \noexpand }

%         \endwrite cannot be grabbed as #2 so we \noexpand it
%   → \weirdb <safe \endwrite token>

\def \weirdb #1 {  \iffalse { \fi } #1   \edef \mycontainendwrite { \noexpand #1 } }

% after it's grabbed once it becomes "unsafe" again
%   → } \endwrite \edef \mycontainendwrite { \noexpand \endwrite } ...

% we need to restore the `} \endwrite` tokens otherwise it's an error

    \write 16{\weird}


\show \mycontainendwrite

\expandafter \let \expandafter \myendwrite \mycontainendwrite

\show \myendwrite  % no output in LuaLaTeX, segmentation fault in pdflatex and XeLaTeX and latex


Basically if you do \write <some number> {<some token list>}, the token list is stored unexpanded in the "whatsit", and when the box is shipped out e.g. with \shipout \box 0 \somemoretokens ..., something similar to \immediate \write <the number> { <the token list> } \endwrite \somemoretokens ... is executed.

(*) As pointed out below, \immediate\write does this as well. Last time I tried it doesn't happen -- probably some mistake on my end.

To store the endwrite somewhere, inject it after the \write (because the content inside is purely expandable, so you can't store it).

Interestingly it creates a segmentation fault. (I wonder if it's exploitable.) it's just a null pointer dereference, on normal environments it's guaranteed to SIGSEGV, so much less "interesting" than the month value arbitrary-location memory read fixed recently.

The output on LuaLaTeX is

> \mycontainendwrite=macro:
->\endwrite .
l.37 \show \mycontainendwrite

> \myendwrite=\outer macro:
l.43 \show \myendwrite

and on the rest is

> \mycontainendwrite=macro:
->\endwrite .
l.37 \show \mycontainendwrite

Segmentation fault (core dumped)

(the segmentation fault is caused by the \let, not the \show.)

Side note for the remaining tokens. You can read texdoc tex and tweak around a bit to figure them out.

First trigger an error that causes the token to be generated. Then "guess" what the resulting following tokens in the input stream will be.

For example, for the \inaccessible token which can be accessed by executing \def followed by a non-control-sequence.

$ pdflatex
This is pdfTeX, Version 3.141592653-2.6-1.40.24 (TeX Live 2022/Arch Linux) (preloaded format=pdflatex)
 restricted \write18 enabled.
**\def a
entering extended mode
LaTeX2e  patch level 1
L3 programming layer 
! Missing control sequence inserted.
 \def a
? I\a{}\show
> \inaccessible=undefined.
 \def a
? I\show
> the letter a.
 \def a

The underlined parts are typed text. The rest are output from pdflatex.

As you can deduce, when \def a is executed, TeX prints an error message, drops the user to the command prompt, then continue as-if \def is just executed and \inaccessible a are following in the input stream.

If you type I\a{}\show here, the content to be executed becomes \def\a{}\show\inaccessible a as you can expect.

This token has .tok value 0x20010001 (or 536936449) in lualatex, but I don't think there's any way to create a token given its tok value anyway.

Alternatively (for academical purpose) it's also possible to use the error hook in Lua to intentionally trigger the error to create the inaccessible token, but if you do this 100 times in the same paragraph TeX will stop.

    luatexbase.remove_from_callback("show_error_hook", "get inaccessible token")
  "get inaccessible token"


\def a{}

Try it online! (link to some online code execution service for a demo and see the result.)

Additional note.

Point 4. The \endtemplate token

If you do the following (similar to the trick to get \endwrite token, but don't need the brace hack)

\def\weird{\expandafter \weirda\noexpand}
\def\weirda #1{\global\edef\container{\noexpand #1}#1}

you can obtain the outer endtemplate token itself, instead of just its meaning. It has the property

  • string representation is also \endtemplate, just like the inner token
  • meaning is \outer endtemplate:, or [unknown command code! (151, 4)]: on LuaTeX. (another bug...?)
  • is outer.
  • is expandable.
  • if \noexpand is applied on it, its meaning becomes \relax, like a normal expandable token.

If it's expanded once, it becomes the inner token which has the property

  • string representation is \endtemplate, as mentioned above
  • meaning is end of alignment template
  • is not outer.
  • is not expandable.

Point 3. notexpanded

As mentioned in the book, its meaning is not equal to the meaning of the normal \relax. It may be noticeable that its meaning can be assigned to another token, the new token would be unexpandable.

Also, this meaning is also obtained when \noexpand is applied on an undefined token.

When \noexpand is applied on a non-expandable token (including the frozen relax), it's not a no-op, but the effect is only shown if you either

  • redefine it to be something else, then the meaning of the token on the input stream becomes \relax
  • apply \show, then \notexpanded is printed on the console. (so \expandafter\show \noexpand\let shows \notexpanded: let even though \let is not expandable)


% ======== assign meaning \a = \noexpand of y
\def\y {}
\expandafter \let \expandafter \a \noexpand \y

% ======== assign meaning \b = \noexpand of z
\expandafter \let \expandafter \b \noexpand \z

% ======== assign meaning \c = \noexpand of \let
\expandafter \let \expandafter \c \noexpand \let

% ======== \a is different from \relax
\ifx\a\relax \errmessage{this cannot happen} \fi

% ======== \a expanded once is different from \relax
\expandafter\ifx\a\relax \errmessage{this cannot happen} \fi

% ======== \a is different from \y
\ifx\a\y \errmessage{this cannot happen} \fi

% ======== \a expanded once remains the same i.e. it's not expandable, unlike the notexpanded token
\expandafter\ifx\a\a \else \errmessage{this cannot happen} \fi

% ======== \a and \b is the same
\ifx\a\b \else \errmessage{this cannot happen} \fi

% ======== \c meaning is \let
\ifx\c\let \else \errmessage{this cannot happen} \fi

  • 1
    Although the \show would segmentation fault as well.
    – user202729
    Jun 27, 2022 at 12:50
  • 1
    The only place where this is ever documented is probably texdoc tex, by the way.
    – user202729
    Jun 27, 2022 at 13:57
  • 2
    Does this segmentation fault happen with the tex engine too? Consider reporting this to the tex-k mailing list ([email protected]). Jun 27, 2022 at 14:42
  • @ShreevatsaR Yes I think? (called with the latex format though.) Will consider later or you can do it as well if you want to.
    – user202729
    Jun 28, 2022 at 2:08
  • 2
    Note: I had used Gmail's "schedule send" to send a report about this to tex-k just in case (pointing to this post), and it's now sent—please feel free to chime in there as needed. Jul 9, 2022 at 18:18

DRF et al. agree the \endwrite access, at least, is a bug and we'll report it to Knuth (eventually). Ultimately it boils down to accessing mem[0], which would be out of range if mem_in>0, as it is in the trip test. Anyway, meanwhile, I added it at https://tug.org/texmfbug/newbug.html#B155endwrite

DRF wonders:

I'm not (yet) convinced that there's no similar trick whereby one could get \let or \futurelet to apply to end_write_token, which would cause trouble when eq_destroy thought the old value was an outer_call (so we'd need the another_null_list fix). But I'm not familiar enough with the intricacies of TeX's mouth and stomach to tell whether \let and \futurelet do have a similar issue; so it might be worth checking with the super-hackers [here] who figured out this sorcery to begin with?

If anyone would like to experiment with that and report, please do. Preferably to [email protected] as well as here.

Also, it's not clear to me if the other things being discussed here are potentially bugs or just strangenesses. If TeX doesn't crash, it's unlikely Knuth would want to change anything, but, hard to say for sure ...

  • 1
    Adapting Phelype's comment a bit, I execute \def\weird{\expandafter\weirda\expandafter{\iffalse}\fi} \def\weirda#1{\expandafter\weirdb\noexpand} \def\weirdb#1{\iffalse{\fi}#1\edef\mycontainendwrite{\noexpand#1}} \immediate\write16{\weird} \expandafter\let\mycontainendwrite\relax \bye which eventually executes \let\endwrite\relax, and TeX correctly report that \endwrite is not assignable ("Missing control sequence inserted. \inaccessible"); similar to how the frozen relax/non-control-sequences are not assignable. I can't get a crash this way.
    – user202729
    Jul 17, 2022 at 1:19

In my experiments, I was only able to produce 1, 2, (partially via \show) 3, 4, but not the last two.

Number 1 can be produced simply

{\tenrm \xdef\tmpa{\the\font}}
\show\tmpa                       % macro:->\tenrm
\expandafter\ifx\tmpa\tenrm y\fi % \ifx equivalent
% \tmpb\tenrm                    % produces error: different shape

Number 2 can be produced and verified to be anomalous using the same tests:

\show\tmpa % macro:->\relax

For 3, I can only see \notexpanded: when using \show on a just \noexpand'd token. Perhaps there are more ways to experiment with them

% > \stuff=\relax.
% <recently read> \notexpanded: \stuff

(this is getting out of hand: there are now at least two anomalous variants of \relax!)

Number 4 can be \futurelet to a control sequence inside an h/valign

\show\tmpa % =\outer endtemplate:

this can be expanded once more to obtain an \endtemplate.

\show\tmpb % =end of alignment template
\show\tmpc % macro:->\endtemplate

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .