Using \typeout behaviour different with \newtoks vs. \def

Question

I feel as I really should know this, but sadly I don't. Why is the output of \typeout{\the\myToken} not the same as having it printed in the PDF as \the\myToken. No problem with a \def, just with something declared via \newtoks.

The PDF shows:

but the console shows

DefaultValue=1-9

MyToken=\DefaultValue

Expansion issue? Well, that is what I thought, but then \expandafter\typeout{\the\myToken} should have worked?

Questions:

• Why is this happening?
• How do I define a \MyTypeout{} (or preferably redefine \typeout{}) macro so that the actual current expanded value is output in the console the same was it would be displayed in the PDF.

Code:

\documentclass{article}

\newtoks\myToken

\edef\CurrentValue{1-9}
\edef\DefaultValue{\CurrentValue}
\myToken={\DefaultValue}

\begin{document}
\typeout{DefaultValue=\DefaultValue}  DefaultValue=\DefaultValue

\typeout{MyToken=\the\myToken}        MyToken=\the\myToken
\end{document}

Answer 1

In typesetting context, the expression \the\myToken is fully expanded and typeset in TeX's stomach. Thus the expected result is 1-9.

Macro \typeout is defined in latex.ltx:

\def\typeout#1{%
  \begingroup
    \set@display@protect
    \immediate\write\@unused{#1}%
  \endgroup
}

In this context (\immediate\write) the expression \the<token register> is expanded once, but the contents of the token register is not expanded further.

From "The TeXbook", "Chapter 20: Definitions (aka Macros)":

Expanded definitions that are made with \edef or \xdef continue to expand tokens until only unexpandable tokens remain, except that token lists produced by \the are not expanded further.

\immediate\write is not \edef or \xdef, thus we also need:

TeX's primitive commands \mark{...}, \message{...}, \errmessage{...}, \special{...}, and \write<number>{...} all expand the token lists in braces almost exactly as \edef and \xdef do. However, a macro parameter character like # should not be duplicated in such commands; [...]

Example:

\edef\x{\the\myToken}
\show\x
> \x=macro:
->\DefaultValue .

\edef\x{\x}
\show\x
> \x=macro:
->1-9.

Thus you need to trigger the expansion twice to get the contents of the token register expanded.

Token register can be used to simulate eTeX's \unexpanded (however the construct will be non-expandable, \unexpanded is expandable). For example, this is used in \g@addto@macro (latex.ltx) that adds some tokens in the second argument to the macro in the first argument:

\toksdef\toks@=0
\long\def\g@addto@macro#1#2{%
  \begingroup
    \toks@\expandafter{#1#2}%
    \xdef#1{\the\toks@}%
  \endgroup
}

Macro #1 is expanded once and the result is put into a token register with the additional tokens in #2. The following \xdef defines the macro with its original definition text with the tokens in #2 appended.

Expansion of token register inside \typeout

The contents of a token register looses their non-expandable behavior if it is touched twice:

\makeatletter
\newcommand*{\exptok}[1]{%
  \expandafter\@firstofone\expandafter{\the#1}%
}
\makeatother

\typeout{\exptok\myToken}

Macro \typeout should not be redefined, the contents of token registers can contain stuff that breaks, if it would expanded (undefined commands, endless recursion, ...). Also it would disturb the usages of \typeout for other, is the tokens where suddenly expanded.