# display full macro expansion

Intrigued by a strange question (how to produce a [La]TeX representation of a Texinfo source file, under the assumption that the Texinfo source file would yield somehow a lower level document in "pure" TeX [presumably \ { } notation] macros), I wonder now how could I redirect the byproduct of macro expansion in TeX's gullet to, say, stdout or console or a text file, before it is fed to the typesetter (the "stomach"). I do not think anything in the \tracing* family of debugging control sequences does that job, as I'd like to see the actual stream of tokens.

I guess I could do that with some arcane LuaTeX callback, but I wonder if a dirty dekTeX macro (with \immediate\write or something like that) would do the trick.

For instance: if we feed pdfinitex with the following minimal input

\pdfoutput 1
\font\myfont ptmr8y
\myfont
Hi!
\end


we may produce a very minimal PDF file (with some errors). I assume these tokens go straight into the typesetter guts, as there are no macros, to produce the PDF output.

If, on the contrary, we run pdftex with the Plain format (loading, presumably, the etex.src macros) with the following input

Hi!
\bye


I conjecture the typesetter shall be fed with at least the following tokens:

\pdfoutput 1
\font\tenrm=cmr10
\tenrm
Hi!
\par \vfill \par \penalty -\@MM \end


How to show (on console, in aux file, anything...) the exact stream of tokens?

Perhaps dekTeX (and derivatives) can't do it; but LuaTeX may have some way.

• Can you at least give an example of a macro call, and what you would hope would end up at the console and what would end up being typeset? Feb 4 at 19:39
• In general this is not possible (the question is a duplicate, I'll find a link) TeX is not like the C preprocessor macros that are processed first expanding to a lower level system, the expansion and execution are interleaved on a token by token level and can not be separated. Feb 4 at 19:54
• tex.stackexchange.com/questions/3174/latex-macro-expander is essentially a duplicate Feb 4 at 19:56
• @DavidCarlisle I don't think I'm asking the same question as the referenced duplicate: the op there wanted a "shallow top layer", in Robertson's words, of macro expansion; enough to replace their custom LaTeX macros with "standard" ones, for journal submission. I am asking for the thoroughly deep layer: the stream of tokens actually sent to the stomach to do the typesetting. Feb 5 at 0:48
• In the case of Hi!\bye processed with pdftex, the instructions \pdfoutput=1 and \font\tenrm=cmr10 \tenrm are not executed: they're already stored in memory when TeX starts up and loads the specified format, plain.fmt in this case. And much more than expanding \bye is done, because the output routine enters into action and several macros are expanded as a consequence. Feb 6 at 14:45

Have a look at the unravel package. It implements TeX in TeX with a few limitations but it does an exceptional job at following the expansion chain for “normal” TeX code.

In you example you mention \pdfoutput=1 and \font\tenrm.... This you cannot obtain at all, not even with LuaTeX because these things are baked into the format which is just a dump of TeX's memory which is then reloaded from disk.

Unfortunately the unravel package is LaTeX-only. This will generate a lot of output.

\RequirePackage{unravel}
\unravel{
\documentclass{article}
\begin{document}
Hi!
\end{document}
}

• Indeed: it was something along these lines. So, it seems that the job only requires eTeX and above, and then somehow mimicking TeX's behavior in the gizzard before reaching the stomach. Neat. Feb 15 at 0:24
1. What is the byproduct of expansion? The product of expansion is unexpandable tokens and/or expandable tokens whose expansion was suppressed for some reason.
2. Assume an expansion-based infinite loop. When is the time to write?
3. The mouth produces tokens. Expansion takes place in the gullet.
4. I don't know of a way of intercepting the token-stream that reaches the stomach feasible in Knuthian TeX.
5. \immediate\write alone does not do the trick: The "argument" of \write is expanded but expansion is not suppressed with tokens where expansion should be suppressed.
E.g., assume \foo is defined to yield bar.
\immediate\write\somewritehandle{\def\foo{rab}} yields that \def bar{rab} is written which seems not to be what you want.
6. The following is a problem when not writing all tokens that reach TeX's stomach (writing all tokens that reach TeX's stomach is not possible in Knuthian-TeX) but only some tokens, e.g., tokens within the argument of \write: Writing a set of (unexpandable) tokens to an external file is not sufficient for ensuring that tokenization at the time of reading/processing that file yields the same set of tokens.
• For example, at the time of writing \escapechar might be changed. E.g. with \escapechar=\x (unexpandable) control sequence tokens are written with leading x instead of leading backslash.
• For example a could have category code 12(other). Writing the token sequence \a, b, c, <space token> then results in the character-sequence \abc<space> being written to the text file. If at the time of reading/tokenizing things from the text-file a has the usual category code 11(letter), then the character-sequence is tokenized as the token \abc. The <space> is not tokenized at all because after tokenizing a control word token the status of TeX's reading-apparatus is set to S(=skipping blanks).
• Explicit character-tokens of category 6(parameter) are doubled during writing. (Usually hashes (#) have category code 6.)
• Depending on the TeX-engine and configuration of TeX's character translation some characters might be written in ^^-notation. If the corresponding part of the resulting file is read/tokenized while ^^-notation is disabled, e.g., while gathering the tokens that belong to the argument of a command \verb, or while gathering the tokens that belong to the "body" of an environment \begin{verbatim}..\end{verbatim}, then Bob is not your uncle any more.
• Actually 6 can be fixed, by reconstructing the whole token list to be executed (with TeX code or Lua code) before executing any of them. Although sure, modification to the engine (or Lua hook, if it's already available) is necessary. and it would take a lot of effort. Feb 5 at 15:29
• This reply looks like a bunch of comments, rather than an answer, so I'll reply in order: 1. Maybe it is the product I am looking for (the byproduct are \message`s and errors and whatnots, conceded); 2. right before feeding the stomach; 3. conceded: it was poor wording (like "byproduct"); 4. what about a callback in LuaTeX? 5. and 6. Conceded. Feb 6 at 4:43
• @jarnosz Ad 3: In TUGboat (Volume 11), 1990, No 2, you find an article "List in TeX's mouth". This did confuse me for a long time until I realized that the title "Lists in TeX's gullet" would probably have been more scary ;-) Feb 6 at 10:38
• That article is perhaps the reason why I misworded my question in the first place o_O Feb 6 at 17:04
• In fact, at first I mistook the "gullet" for a gizzard, and I believed TeX was more like a hen than a lion; then I reckoned the intricacies of the interactions of this complex digestive tract, and now I believe TeX is a brainless cow ^_^ Feb 6 at 17:34