# (Why) Can't we get 'fully expandable' versions of every command?

I'm just starting to figure out what 'expandable' means in LaTeX, and why it is useful to have commands that expand as far as possible. In particular:

• If you want to \typeout some string to the log file, you need to be able to expand all the way to it.
• If you want to do any kind of unit-testing (like with qstest), you need to be able to expand at least down to the strings you want to compare.

I've been reading a lot about this topic. As I understand it, a command is not expandable if it changes the 'state' in some way, i.e., uses \def, \let, etc. In particular, I've also read the question "Why isn't everything expandable". But my question is less theoretical than that one. I'd like to know why I haven't encountered any 'solutions' yet.

The thing is, 'making everything expandable' is conceptually so simple. TeX is perfectly capable of typesetting the result of 'unexpandable' commands, simply by performing the state changes in the execution processor. The point being, TeX does eventually arrive at the final string.

(Edit: Please note that I'm talking about textual characters here. I understand that one cannot capture the notion of, e.g., an image, flexible whitespace or a state-manipulation in a string.)

It seems inconceivable to me that we can't somehow intercept it and use it. I would have expected someone to have created a command like this:

\getresult { <expr> } { <result-macro> }


It's not expandable, but we can still get our hands on the fully expanded result in order to log it... compare it... what have you.

Questions like this one suggest to me that this has not been accomplished.

Why?

Ok... I'm willing to cheat. What if we spawn an extra TeX process to fully evaluate the expression and (almost) typeset it. Can we get the string that way? Doesn't \scantokens use a similar trick to overcome another inherent TeX limitation (that catcodes are fixed once a token is read)?

Can it be done?

-
I suspect you are mixing up what you see on a typeset page (probably mainly letters) with what TeX 'sees' (boxes, kerns, glue, font selection, ...). For example, what doe would the full expansion of something like \hbox{ABC\hskip 1 cm DEF} be? For example, take a look at the \tracingall output of a typeset page. –  Joseph Wright Nov 23 '12 at 15:29
Yes, of course. I was talking about capturing characters. I will edit my question to clarify this point. ### I do know that when I generate a document with TeX, the resulting text is not lost. It doesn't become a 'bitmap'. I know, because I can still select and copy the text from my document viewer. –  mhelvens Nov 23 '12 at 15:33
TeX doesn't really have a concept of "textual characters". You're talking about box contents here. In fact, with modern Unicode-aware systems it might even make some sense to try to recover the "text content" of a box, but TeX doesn't offer that out-of-the-box. Maybe it could be programmed in LuaTeX? –  Stephan Lehmke Nov 23 '12 at 15:40
Would you please add an example of what you really need? –  egreg Nov 23 '12 at 15:44
@egreg I wanted to keep this question more general, partly out of intellectual curiosity. Since it now seems that the answer is no, I will follow up with questions about my specific problems. –  mhelvens Nov 23 '12 at 16:01

The problem is that this assumption is basically false:

The point being, TeX does eventually arrive at the final string

It is not like C and its macro pre-processor cpp where all the expansion happens first resulting in an expanded version of the C file which can be passed to the C compiler (or intercepted for debugging or other reasons). In TeX non expandable operations and expansion are inextricably interwoven.

The classic example is

    \setbox0\hbox{hello}\the\wd0


\wd0 is expandable and expands to the decimal expression of the width of box 0. However you can not expand it until you have dome the non-expandable operation of typesetting text into the box.

so given

 \def\foo#1{setbox0\hbox{#1}\the\wd0}


You can not have an expandable version of the command \foo.

The distinction between expandable and non expandable is often arbitrary. So the answer is that in TeX it can not be done. It's like saying my name is David. There are alternative histories where different things may have been true but we have the system we have.

Consider arithmetic:in classic TeX to print one more than some value you have to go

\advance\mycount by 1 \the\mycount


and that is a non-expandable operation. In etex you can go

\the\numexpr\mycount+1\relax


which is an expandable operation. There is no reason that one can argue for why one version of arithemtic is expandable and the other is not. It just is. If someone is extending TeX (eTeX, pdftex xetex, tex part of luatex ) then any new primitive that is added has to be classified as expandable or not expandable as the TeX programming language requires that distinction. Commands are expandable or non expandable because the person who designed them classified them that way.

As noted in the comments, a possible workaround of writing out fragments to a sub-process running a different instance of TeX can not help, even if \write were expandable. The result of evaluating any argument is highly context dependent (it may involve references, definitions within the document or reference the time the job started) it would be impossible in general to generate the same text in the sub process as generated in the original process.

It's probably worth noting that the term in quotes in the question "fully expandable" isn't well defined (or definable) which is part of the reason for the confusion around the topic of expansion. As discussed in an earlier question Advantages and disadvantages of fully expandable macros a better term is "safe in an expansion only context". Commands that are safe in such expansion only contexts as \edef or \write include character tokens and the \relax primitive mentioned in the comments.

-
@mhelvens A second TeX process will not solve the expandability problem, at least without LuaTeX, as \write18 is not expandable. –  Joseph Wright Nov 23 '12 at 15:47
@JosephWright yes of course. You could just declare everything is expandable but then you would have to think of a new word for the distinctions that are necessary. It depends a bit what the intention is The OP said "expansion problems TeX-programmers are having" But I don't think making things expandable actually solves any of those problems, it is perhaps more accurate to say that people have problems programming TeX. –  David Carlisle Nov 23 '12 at 16:18
Even if \write were expandable I doubt there are many cases where it would help for the reasons I gave. TeX is almost always highly context dependent, It is almost never possible to execute a fragment in isolation (which is why we always ask for complete documents as examples in questions, without that code fragments are often unusable) –  David Carlisle Nov 23 '12 at 16:20
@DavidCarlisle (in response to your last two comments) That is, of course, only partly true. There are many cases where 'making non-expandable commands expandable' would be both quite safe and tremendously useful. You're of course quite right that it could be very difficult or damaging in the general case. ### The problem is finding a suitable dividing line. And I think TeX may currently be drawing it too much to the safe side. ### In any case, again, your answers and comments have, as always, been enormously useful. Thanks! (Also to Stephan. I should definitely look at LuaTex some day.) –  mhelvens Nov 23 '12 at 16:26
The main hurdle is that a) in TeX all assignments are non-expanable and b) that to operate (in an exapandable manner) on typeset text you first have to assign it to a box. So whenever typsetting is involved you are having to jump to the non-expandable side and there is no way to change this in the model. And that means anything that requires measuring of material, for example is doomed –  Frank Mittelbach Nov 23 '12 at 16:29

Understanding "expandable" as TeX does is tricky, because there are really two kinds of result for a TeX command sequence (something such as \foo, which is commonly called a "macro", but which I won't because that has the baggage of macro expansion, which I'm trying to demystify rather than further confuse). I'll be using made-up terms for these.

• The first kind of result is input, and this is what happens after "expanding" \foo. This just replaces \foo with other tokens that can be further scanned by TeX.

• The second kind of result is output, and this is what happens when the result is no longer scanned by TeX. This has two sub-types:
(a) Internal output. This is what the programming constructs like \def and \let do, as well as other assignments like \count0=1 that aren't exactly implemented by a control sequence.
(b) External output. This is what is typeset. However, even if nothing ever made it to the page, it is still placed in a box in memory, and the contents of the box are rendered with fonts and explicit spacing.

TeX's operation is a long crawl through a stream of input tokens that are either converted to external output (if they are directly typesettable material like letters or other font symbols, or are boxes and such), to internal output, if they are assignments of some kind and thus "save" their result for later use, or to more input, if they simply expand right back into the input stream. These processes can be mixed, of course, such as when you assign to a box register (as internal output) a box that was produced as external output.

The process of expansion is blind: it does not interact with the mechanisms that produce external output. These are the measurement and placement mechanisms, so the example of \setbox0=\hbox{hello}\the\wd0, in passing through the external output mechanism to do the width computation, is necessarily non-expandable, because width does not exist at the input level. (As it happens, it also has to pass through the internal output mechanism to save the box, but that is dictated only by the design of TeX that forbids you from writing \the\wd\hbox{hello}.)

Now, you ask for the "final result" of something that is typeset to be available as input. I think what you mean is the following:

You want to circumvent the restriction on expanding internal output, i.e. you want \def\foo{hello}\foo to be converted to hello. This sounds like a reasonable request that, as David Carlisle said, is simply impossible. TeX designates \def to be non-expandable and that's it. One wonders (I once did) why \def does not "expand" to nothing as well as making the definition, and there's no reason I know of, but the fact is that as things stand, it passes through the internal output mechanism and therefore does not directly manipulate the input stream.

If I may philosophize some more, what you are requesting is I think a common way of "doing it wrong". You want to issue a programming construct that results in further input that is also valid as a programming construct for producing external output. Since the two languages, though intertwined, are actually different, this is impossible. That doesn't mean there aren't equivalent ways of writing your input construct so that it creates what you want. A common workaround is to do a lot of non-expandable stuff that repeatedly builds a macro \result, and then its contents are available expandably afterwards (this is what the previous examples do, actually).

-

Maybe David's answer can be complemented by a non-TeX perspective.

As TeX macro processing is comparable to a programming language the notion of expandable versus non-expandable does exist in computer science but with different wording.

Essentially a fully expandable macro or function is a pure function, i.e. a function without side effects that does not depend on anything but it's arguments. In pseudo-C:

int twice(int x)
{
return 2*x;
}


For a given string it does not change the output whether we replace the function by it's return values: twice(4) is equal to 8. Now we take a slightly modified function and end up with something that is inherently non-expandable:

int twice_current_page()
{
return 2*current_page;
}


twice_current_page() cannot be replaced/expanded in the document. The only way to get the result is to execute everything up to this point and then execute the function body.

-
This is very far from the usual understanding of expandable wrt. TeX. –  Stephan Lehmke Nov 23 '12 at 17:25
@StephanLemke: Yes, I tried with C as an example language but I am not sure if it translates nicely back to TeX, feel free to improve. –  Alexander Nov 23 '12 at 17:49