1

The following plain TeX code is modeled after a similar line of code in the pgf/tikz source code (near the end of the the \pgfoo@inherit@methods macro in <pgf/tikz installation dir>/modules/pgfmoduleoo.code.tex):

\edef\c{%
    \noexpand\let\expandafter\noexpand\csname a\endcsname%
    \expandafter\noexpand\csname b\endcsname%
%}

When I print \c's \meaning I get:

macro:->\let \a \b

This is not what I expect. What I expect is

macro:->\noexpand \let \expandafter \noexpand ... \csname b\endcsname

In other words, I expect the exact same list of tokens as was used to define \c. The reason for this expectation is that \edef is supposed to expand only macros, but none of the tokens in \c's definition is a macro.

Why is \c's \meaning the way it is?

  • 3
    \noexpand, \csname and \expandafter are expandable – egreg Aug 16 '17 at 21:25
  • @egreg: So what is the definition of an expandable token? Why is \noexpand expandable, but \let isn't? How can I tell? – Evan Aad Aug 16 '17 at 21:27
  • From the TeX source. obligatory read --> tex.stackexchange.com/questions/12482 – percusse Aug 16 '17 at 21:31
  • Any expandable token is expandable, the others aren't. ;-) The TeXbook lists all expandable primitives. – egreg Aug 16 '17 at 21:35
  • BTW, "expandable" is a really poor name: it sounds like "can be expanded", but rather it means "will be expanded (usually)". That is, an expandable token is something such that, when TeX encounters it as it passes through its input stream of tokens, it calls an internal expand function. (In the normal case, though things like \expandafter can alter this behaviour.) – ShreevatsaR Aug 16 '17 at 21:40
9

Macros, that is tokens defined with \def, \edef, \gdef or \xdef are expandable. Every token \let to an expandable token is expandable.

Primitives can be expandable; the expandable primitives are

  • \csname
  • \else
  • \expandafter
  • \fi
  • \if... (any primitive conditional)
  • \input
  • \meaning
  • \noexpand
  • \number
  • \or
  • \romannumeral
  • \string
  • \the

e-TeX adds some more conditionals and the expandable primitives

  • \detokenize
  • \eTeXrevision
  • \scantokens
  • \unexpanded
  • \unless

Note that e-TeX also adds \protected; a macro defined with \protected\def will not be expanded in the general texts for \edef or \write.

The pdftex, xetex and luatex engines add more expandable primitives; check their documentation.

In the code you write, the first \noexpand does nothing at all. Indeed, the role of \noexpand is to make the following token equivalent to \relax at the next usage (hence unexpandable); the expansion of \noexpand is empty, after having performed its duty. However, \let is not expandable to begin with, so it isn't touched during \edef.

Similarly, \expandafter expands to nothing after having expanded the token after the following one (if expandable at all).

Therefore the code in your \edef\c becomes, in turn,

\let\expandafter\noexpand\csname a\endcsname\expandafter\noexpand\csname b\endcsname

(\let is equivalent to \relax)

\let\noexpand\a\expandafter\noexpand\csname b\endcsname

\let\a\noexpand\b

(\a is equivalent to \relax)

\let\a\b

(\b is equivalent to \relax)

When \c is expanded, \let \a and \b are not equivalent to \relax any longer.

  • Thanks. Can you please also explain why \edef\c{\let\a\b} is not equivalent to the original expression, and produces an error, even when \b is defined? – Evan Aad Aug 16 '17 at 21:47
  • @EvanAad because \a and \b would get expanded inside \edef. – Skillmon likes topanswers.xyz Aug 16 '17 at 22:02
  • …provided, of course, that they are expandable, and not be \let equal to unexpandable tokens (addition to @Skillmon’s comment) – GuM Aug 16 '17 at 22:04
  • Since you gave a non-exhaustive list of expandable control-sequences, could you please complement it with a list of some popular non-expandable control-sequences? – Evan Aad Aug 17 '17 at 6:58
  • Thanks for the edit. An assignment of a value to a register is not expandable, right? Nor is \relax? – Evan Aad Aug 17 '17 at 18:09
4

OK, as requested, here is a run-through of what happens in the TeX program, in the case of \def and \edef respectively. For this question it adds absolutely nothing over the basic point (that \noexpand, \expandafter etc. are indeed “expandable”), but it may be interesting to you anyway.

You will want to refer to the TeX program as you read the following.

What TeX's main_control procedure (section 1030) does is to read a token (get_x_token) in (essentially) a loop, and decide what to do. The inner loop is tightly optimized for regular characters in horizontal mode, but things like \hrule or \def are not part of it (→ section 1045), and in fact \def and \edef are part of mode-independent processing (→ section 1210), and result in a call to the procedure prefixed_command (section 1211). This procedure is common to many primitives/commands, but for both \def and \edef, it is called with:

  • cur_cmd = [a code indicating "def"], and
  • cur_chr = [0 for \def, 1 for \gdef, 2 for \edef, 3 for \xdef]

(As you may guess from these four numbers, Knuth later checks "odd(cur_chr)" to decide whether the current definition should be global or not, and "e = (cur_chr >= 2)" to decide whether to expand or not!)

Anyway, in this prefixed_command procedure (section 1211), the execution falls into section 1217 then 1218, which is:

§1218 def

Suppose your input had \def\foo{\bar} or \edef\foo{\bar}. Then in the above, get_r_token; p ← cur_cs; would set p to (basically) \foo, and then scan_toks(true, e) is called.

Here, of the two parameters to scan_toks, the first one (macro_def, here passed as true) says that the token list to be scanned is that for a macro definition (as opposed to the token list for a \mark, \output, \message, \everypar, etc.), and the second boolean (xpand, here passed as e) decides whether to expand or not.

So let's dive into scan_toks (section 473).

§473 <code>scan_toks</code>

First, as we're in macro_def, it goes into ⟨Scan and build the parameter part of the macro definition⟩, section 474. In our example (parameterless macro), all that happens is that get_token gets token begin-group character {, and therefore this part simply ends with end_match_token being added to the token list. The token list for the actual body of the definition is scanned in section 477.

This one basically gets tokens one-by-one and adds them to the token list being built (for the body of the definition), with the crucial difference that in the case of \edef or \xdef, the tokens get expanded as we go along:

§ 477 scan body token list

  • In the case of \def\foo{\bar} there is no complication of expansion and this part of the token list contains basically just the token \bar; so after scan_toks returns, the definition of \foo is saved as the token list containing two tokens, end_match and \bar.

  • In the case of \edef\foo{\bar}, this part (remember we're still inside scan_toks) reads tokens from the input stream (currently containing \bar) one at a time, each time expanding any token it sees. Specifically, the expansion here happens in section 478's call to the procedure expand (section 366). The way this procedure works is that each time it is called, it (essentially) simply destroys the first token in the input stream, and replaces it with the “expansion” (whatever is appropriate) of the removed token. This way, the next calls to get_token (say) will pick up the replaced tokens.

As you can see, here “expand” / “expansion” / “expandable” means basically anything that has a replacement (i.e. should be changed somehow), which does not mean only macros, and includes things like \noexpand and \expandafter:

§ 366, expand, part 1 § 366, expand, part 2 § 366, expand, part 3

(Don't try to directly infer the expandable TeX primitives from the names of the variables code; instead see egreg's answer. For example, the primitives \number, \romannumeral, \string, \meaning, \fontname, \jobname are all expandable, because they all result in the command code convert that is in the list in the code above.)

  • Thanks. It's so helpful to have a guided look under the hood at the actual implementation. Do you get all this by simply reading the source-code, or do you also have access to a debugger for the TeX program, so you can set breakpoints and watch the values of internal variables during runtime? – Evan Aad Aug 18 '17 at 10:59
  • 1
    @EvanAad This was all newly gained and hard-won knowledge :-) I built TeX Live's tex from source with -g and then the binaries without strip, then stepped through the program execution in gdb. The source code generated by web2c is quite hard to read. For example where the woven WEB program (as in screenshots above) contains “ if every_job ≠ null then begin_token_list(every_job, every_job_text); ”, the C code contains if ( eqtb [25064 ].hh .v.RH != -268435455L ) begintokenlist ( eqtb [25064 ].hh .v.RH , 12 ) ; – ShreevatsaR Aug 18 '17 at 14:52
  • … so I haven't yet figured out the equivalent of watching internal variables. But even though the code is rather different, with both the woven TeX program (texdoc tex) and the source code open, there are enough clues to manually (with some pattern-matching) follow execution in terms of the readable TeX program, by seeing which instructions are getting executed in the less-readable C program. What a function returned or what a token list contains is just inference/guess currently, but perhaps one can write some gdb helper utilities to print out variables in a readable form if type is known. – ShreevatsaR Aug 18 '17 at 14:57
  • What do you mean by "... and then the binaries without strip"? If you'd already built tex from source, why couldn't you just use the binaries generated by this build? – Evan Aad Aug 18 '17 at 16:03
  • @EvanAad Yes you are right; it's just that the Build script distributed with texlive immediately runs strip after generating the binaries. So you have to make sure that step doesn't run (or you use the binaries from before that step); that's all I meant. (See here and here.) BTW if you'd like to try it yourself, note that you need about 7.5 GB. – ShreevatsaR Aug 18 '17 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.