# \expandafter with \uppercase

Consider the following plain TeX manuscript:

\expandafter\expandafter\expandafter\meaning%
\expandafter\uppercase\expandafter{a}%
\bye


When compiled with pdftex, the resulting pdf consists of the following text:

\uppercasea

However, I expect the output text to be

the letter A

Here's why. To the best of my understanding, TeX is supposed to process the manuscript as follows.

1. The first \expandafter is removed from the input stream, and the originally second \expandafter is removed and pushed on an empty stack.
2. The originally third \expandafter is removed from the input stream, and \meaning is removed and pushed on the above-mentioned stack.
3. The originally fourth \expandafter is removed from the input stream, and \uppercase is removed and pushed on the stack.
4. The last \expandafter occurrence is removed from the input stream, and { is removed and pushed on the stack.
5. a is expanded, but since there's nothing to expand, nothing happens, and a remains on the input stream.
6. The stack is popped until empty, and each popped token is prepended to the input stream. The resulting input stream is \expandafter\meaning\uppercase{a}\bye.
7. \expandafter is removed from the input stream, and \meaning is removed and pushed on the now empty stack.
8. \uppercase is expanded. This transforms the input stream to A\bye.
9. The stack is popped until empty, and each popped token is prepended to the input stream. The resulting input stream is \meaning A\bye.
10. \meaning is expanded, which entails printing

the letter A

to the dvi file. The input stream is then simply \bye.

11. pdftex stops.

* The present question is not the same as this one, despite the similar titles.

• @EvanAad You can do expandable case changing but it's non-trivial: see expl3's \tl_upper_case:n and \str_fold_case:n, etc. – Joseph Wright Aug 23 '17 at 12:49
• See the last paragraph in tex.stackexchange.com/a/44459/4427 (it's about \lowercase, but \uppercase is the same). – egreg Aug 23 '17 at 13:05

Step 8 is wrong. \uppercase is not expandable. Try \edef\test{\uppercase{a}} \show\test to see it.

After reading this question, I started writing a “TeX debugger” that may help illuminate matters like this (also stimulated by an earlier question). For now it simply dumps TeX's internal state at various points, and displays it. In case there's any interest, I present its current (very crude) version here. (I think it's about say 2% done, though I'm probably underestimating the amount of remaining work.)

Screenshots/recording:

Unfortunately my UI skills aren't great and it's not done yet, so it's not self-explanatory and needs a commentary. :-) What you can see at the link is what TeX does on the following input file (basically equivalent to the one in the question):

\expandafter\expandafter\expandafter\meaning\expandafter\uppercase\expandafter{a}
\end


Let's label the \expandafters in the input by subscripts so that we can refer to them better in the commentary that follows (the subscripts here are just added by me for the commentary; they're not part of TeX's processing):

\expandafter₁\expandafter₂\expandafter₃\meaning\expandafter₄\uppercase\expandafter₅{a}


So, with this input line, this is what TeX does:

1. ↳ Reads \expandafter₁, and (as it's expandable) calls the internal function expand to start acting per \expandafter₁. Link. At this point, tokens yet to be read: \expandafter₂\expandafter₃\meaning\expandafter₄\uppercase\expandafter₅{a}

2. ↳ ↳ Acting per \expandafter₁, saves token \expandafter₂ as the token to be “expanded after”, reads \expandafter₃, and (as \expandafter₃ is again expandable) calls expand again to start acting per \expandafter₃. Link. At this point, tokens in the function stack: \expandafter₂. Tokens yet to be read: \meaning\expandafter₄\uppercase\expandafter₅{a}.

3. ↳ ↳ ↳ Acting per \expandafter₃, saves token \meaning as the token to be “expanded after”, reads \expandafter₄, and (as \expandafter₄ is again expandable) calls expand again to start acting per \expandafter₄. Link. Tokens in the function stack: \expandafter₂ and \meaning. Tokens yet to be read: \uppercase\expandafter₅{a}.

4. ↳ ↳ ↳ ↳ Acting per \expandafter₄, saves \uppercase as the token to be “expanded after”, reads \expandafter₅, and (as \expandafter₅ is again expandable) calls expand again to start acting per \expandafter₅. Link. Tokens in the function stack: \expandafter₂, \meaning, and \uppercase. Tokens yet to be read: {a}.

5. ↳ ↳ ↳ ↵ Acting per \expandafter₅, saves { as the token to be “expanded after”, reads a, and (as a is not expandable), does not recurse, and puts these tokens back on the stack. (Done acting per \expandafter₅.) Link. Tokens in the function stack: \expandafter₂, \meaning, \uppercase. Tokens to be read: {₁, a₁₁, }. (Here the subscripts denote category codes; they are part of TeX.)

6. ↳ ↳ ↵ The \uppercase that was saved in Step 4 is now pushed back to the front of the input stream. (Done acting per \expandafter₄.) Link. Tokens in the function stack: \expandafter₂ and \meaning. Tokens to be read: \uppercase, {₁, a₁₁, }.

7. ↳ ↵ The \meaning that was saved in Step 3 is now pushed back to the front of the input stream. (Done acting per \expandafter₃.) Link. Tokens in the function stack: \expandafter₂. Tokens to be read: \meaning, \uppercase, {₁, a₁₁, }.

8. ↵ The \expandafter₂ that was saved in Step 2 is now pushed back to the front of the input stream. (Done acting per \expandafter₁.) Link. Tokens in the function stack: None. Tokens to be read: \expandafter₂, \meaning, \uppercase, {₁, a₁₁, }

9. ↳ Reads \expandafter₂, and (as it's expandable) calls the function expand to start acting per \expandafter₂. Link. Tokens to be read: \meaning, \uppercase, {₁, a₁₁, }.

10. ↵ Acting per \expandafter₂, saves the token \meaning as the token to be “expanded after”, reads the token \uppercase, and (as \uppercase is not expandable), does not recurse, and puts these tokens back on the stack. (Done acting per \expandafter₂.) Link. Tokens to be read: \meaning, \uppercase, {₁, a₁₁, }. [Note: This is the step where the analysis in the question was wrong; you can see this in the UI without having to know beforehand whether \uppercase was expandable or not: you can tell it's not, because expand was not called. Or at least, you could tell if the UI were intuitive enough, which it isn't at the moment, but the data is there.]

11. ↳ Reads \meaning, and (as it's expandable) calls the function expand to start acting per \meaning. Link. Tokens to be read: \uppercase, {₁, a₁₁, }.

12. ↵ Acting per \meaning, reads the next token \uppercase, and pushes corresponding tokens to the front of the input stream. (Done acting per \meaning.) Link. Tokens to be read: \₁₂, u₁₂, p₁₂, p₁₂, e₁₂, r₁₂, c₁₂, a₁₂, s₁₂, e₁₂, {₁, a₁₁, }. These ultimately get typeset as the corresponding characters (which is why you see \uppercasea in the PDF). (After this TeX processes the \end leading to the output routine and so on.)

I “generated” the above commentary by looking at the data displayed, and I hope the above is clear, but my ultimate goal is that it should not be needed and the UI should be self-explanatory.

I don't know if anyone else will be interested in this “debugger”. I plan to set it aside and not work on it for the next few weeks at least. Then when/if I return to it, there are many limitations to remove before it can be worthy of the name:

• Improve the UI and present all the information in a better way.
• Adapt it to other TeX engines (this should be relatively easy, at least for pdflatex and xelatex… possibly even for lualatex but that already has many callbacks (and growing) so this kind of thing may be less necessary there).
• Cover other parts of TeX: right now it only illuminates a very small part of TeX: macro (and a bit of other) expansion (i.e., calls to the expand function).
• Make it faster: right now it slows down TeX by a huge factor. (A good thing that TeX is so fast, so this is not a problem.) For example, this could be done by making it possible to “debug” only part of TeX's execution rather than its whole lifetime.
• [Most challenging] It is called a “debugger” but it only shows what TeX is doing and does not let you modify it: would be good to be able to.

In the meantime if anyone else is interested or would even like to contribute/take over, all the code is available on Github. It's basically a Python script to be run from inside gdb running a debug build of tex (for dumping the data) and a HTML/JavaScript “app” (for loading and interactively displaying this dumped data).

• +1 maybe you will be interested by unravel package of @BrunoLeFloch – user4686 Sep 13 '17 at 6:20
• @jfbu Thanks, that (unravel) was in fact an inspiration (one of the) to start working on this! The goal is roughly the same, but the approach is different: instead of implementing (much of) TeX in TeX (wow!) as unravel does and risking some differences, I thought it may be interesting to keep TeX untouched, and collect information “on the side”. Also, there are more UI possibilities when not restricting oneself to terminal output. Of course at the current stage unravel is still probably clearer. :-) – ShreevatsaR Sep 13 '17 at 7:05