When to use \edef, \noexpand, and \expandafter?

Question

I'm quite happy hacking TeX macros and cobbling together bits and pieces from different style files to suit my own ends, but I have a suspicion that my resulting hacks are not quite as elegant as they could be. In particular, with regard to when to expand and when not to expand macros.

A common occurrence for me is defining a meta-command that defines a whole slew of sub-commands. So the names of the sub-commands will contain parameters depending on the parameter passed to the meta-command, and the contents of the sub-commands will also vary a little depending on what the meta-command got. Often it won't be a direct substitution but rather a "if #1 is a do \this else do \that", but \this and \that need expansion at define time, not call time.

Here's a very simple example just involving direct substitution:

\def\cohtheory#1{
\expandafter\newcommand\expandafter{\csname #1func\endcsname}[1][*]{%
\MakeUppercase{#1}^{##1}}
}

Sometimes I worry that my commands are more complicated than they need be. For example, if I want to call a command with two arguments and the arguments expand before the command, here's how I've coded it:

\expandafter
  \expandafter
  \expandafter
  \command\expandafter
            \expandafter
            \expandafter
              {\expandafter
                \argone
                \expandafter
              }\expandafter
              {\argtwo}

So I'm looking for guidance on when and how to control expansion in defining macros. I strongly suspect that such cannot be given in a simple answer, so to make this a focussed question, let me phrase it thus:

Where's a good reference for writing TeX macros that includes advice on how to best deal with how to handle expansions?

Of course, if anyone can formulate some advice in short answer, I'd be only to happy to read it.

(Note: this was partially motivated by juannavarroperez's adaptation of my answer to this question where over 20 \expandafters got condensed down to just 1!)

Answer 1

Expansion is a complicated area of TeX programming. I'll try to explain the key primitives involved first, then try to come up with some examples.

The \expandafter primitive expands the token after the next one. So

\expandafter\def\csname an-awkward-name\endcsname

will expand \csname before \def. So after one expansion the above turns into

\def\an-awkward-name

which will then do its thing. Life becomes more complex when you want to step further ahead, and it soon becomes very hard to track what is going on.

The \edef> primitive does a full expansion of what is given as its argument (in contrast to \def, which simply stores the input). So

\def\examplea{more stuff}
\edef\exampleb{Some stuff \csname examplea\endcsname}

will expand the \csname name\endcsname to \examplea, then expand that to leave a final definition of \exampleb as 'Some stuff more stuff'.

Now, \noexpand comes in by preventing \edef from doing an expansion of the next token. So if I modify my above example to read

\def\examplea{more stuff}
\edef\exampleb{Some stuff \expandafter\noexpand\csname examplea\endcsname}

then what will happen is that the \edef will execute the \expandafter, which will turn the above effectively into

\def\examplea{more stuff}
\edef\exampleb{Some stuff \noexpand\examplea}

Now the \noexpand will operate (disappearing in the process), leaving the definition of \exampleb as 'Some stuff \examplea'.

We can use this ability to cut down on \expandafter use, but there are a couple of other things to know. First, e-TeX includes an additional primitive \unexpanded, which will prevent expansion of multiple tokens. Secondly, there are various special cases where you don't need quite so many \expandafter statements. A classic example is from within \csname, as this will do expansion anyway. So you'll see things like

\csname name\expandafter\endcsname\token

which will expand \token before \name.

Back to your example. In the first one, there isn't much to do: as the entire point is to have a dynamic name (#1), doing an \edef at point-of-definition doesn't really make sense. The closest one can get is something like

\edef\cohtheory{%
  \noexpand\newcommand\expandafter\noexpand\csname foofunc\endcsname[1][*]{%
  \noexpand\MakeUppercase{foo}^{##1}}%
}

What will happen here is that \newcommand and \MakeUppercase will be protected from expansion, and the \csname will only expand once. (Tokens which don't have an expansion don't need protection, which is why things like '[1]' are simply included as is.) Of course, this is something of a 'toy' as all it does is create a fixed \foofunc.

For your second example, you could instead do this:

\begingroup
  \edef\temp{%
    \endgroup
    \noexpand\command
    {\unexpanded\expandafter{\argone}}%
    {\unexpanded\expandafter{\argtwo}}%
  }
\temp

I'm using a couple of extra ideas here. First, the group is used so that \temp is not altered anywhere other than where I'm using it. The \endgroup primitive will do nothing inside the \edef, and so will still be there to close the group when \temp is used. Secondly, \unexpanded works like a toks, and so will respect the \expandafter after it but before the {. This cuts down on an unnecessary \expandafter.

There are more wrinkles to this, and often there are several equally-efficient and clear methods. You are best off posting specific examples, and seeking advice on how they might be achieved.

Answer 2

One difference of \expandafter and \edef is their behaviour towards protected macros.

eTeX provides the prefix \protected which can be used before \def and friends to define a protected, i.e. "robust" macro which doesn't expand inside an \edef context (like in \write). However, \expandafter does expand such a macro.

See the following example (works with eTeX and LaTeX):

\protected\def\pempty{}

\edef\withedef{x\pempty x}
\expandafter\def\expandafter\withexpandafter\expandafter{\expandafter x\pempty x}

\tt
\meaning\pempty.

\meaning\withedef.

\meaning\withexpandafter.

Gives:

\protected macro:->.
macro:->x\pempty x.
macro:->xx.

---

There are also the situations where TeX is expanding tokens, like after & and \cr inside a \halign. Here TeX stops when finding a protected macro without expanding it. However, if TeX is expanding tokens while in number reading mode like for \ifnum protected macros are also expanded.

Answer 3

For the particular problem of creating control sequences dynamically, I suggest you use something like

\def\csarg #1{%
    \begingroup
    \expandafter
    \endgroup
    \expandafter #1\csname
}

You use it like

\csarg\mycommandbuilder <whatever> \endcsname

You can even use it like this

\csarg\mycommandbuilder <whatever> \expandafter\endcsname
    \csname <whatever2> \expandafter\endcsname
    \csname <whatever3> \endcsname

However, take care with the spaces. The above code constructs control sequences that have spaces in their names!

Answer 4

Sometimes having a few helper macros makes the code much more readable. For example, in ConTeXt, typically your first macro will be written as

\def\cohtheory#1%
   {\setvalue{#1func}{\dodoubleargument\docohtheory[#1]}}

\def\docohtheory[#1][#2]%
   {#1^{\ifsecondargument #2 \else * \fi}}

where \setvalue is roughly equivalent to the expandafter newcommand bit. (ignore the difference in the manner in which optional arguments are handled).

The second macro will typically be written as

\expanded{\command{\argone}{\argtwo}}

where \expanded fully expands its arguments. \expanded is defined roughly in the same manner as Joseph's \temp macro, but uses \xdef instead of \edef.

Answer 5

To make this answers complete, one has to know, that TeX uses scopes, that are regions, where the defined macros are know. If you leave that region, the macro (and of course its content) will be destroyed. Lets make an example:

\def\a{foo}
{
  \def\a{bar}
  The actual content of \verb|\a| is \a
}
The actual value of \verb|\a| is \a

will produce

The reason is due to the fact, that the braces {} form a new scope. Within this scope, the macro \a is local, so to say: it is a quite new macro which stores different content compared to the still existing macro with the same name but resting in a different scope. Therefore, the first call of \a results in the content "bar". Afterwards the scope area ends, the local variable is destroyed and the macro, that belongs to the surrounding scope is "restored". The second call of \a from the outer scope will present the content, that was initially stored in \a: "foo".

If you need to work on the same value in different scopes, you have to use a global variable. Here is another example, to make the point clear:

%% Second example with \b
{
  \global\def\b{baz}
  %% Do some things on \b
}
The actual value of \verb|\b|, defined in a not more existing scope is
still `\b'.

And the result:

Instead of the above more literal code, I could also have used the shorter form with \gdef:

%% Second example with \b
{
  \gdef\b{baz}
  %% Do some things on \b
}
The actual value of \verb|\b|, defined in a not more existing scope is
still `\b'.

The result is still the same:

Please see the other (real good!) answers about the difference between the normal \def command, and its companion, the expanding version \edef. You can, of course, play the game with expanding macros in the local scope or in the global scope:

%% Third example with \edef and \global\edef
%% Define the variable \foo to show, where and why this is happening.
%% First: the value before the new scope
\def\foo{before}
\def\a{\foo}
\edef\b{\foo}
\def\c{}
\def\d{}
{
  %% Second: within the scope
  \def\foo{within}
  \def\c{\foo}
  \edef\d{\foo}
  \global\def\e{\foo}
  \global\edef\f{\foo}
}
%% Third: after the scope was left, now again in the global scope
\def\foo{after}

Results 
\begin{tabular}{@{} cc @{}}
  \toprule
  \multicolumn{1}{@{} H}{Variable} & \multicolumn{1}{H @{}}{Content} \\
  \midrule
  a & \a \\
  b & \b \\
  c & \c \\
  d & \d \\
  e & \e \\
  f & \f \\
  \bottomrule
\end{tabular}

It should not be surprising, that the variables \c and \d are still empty, as they have not been defined \global, when they were manipulated in the scope:

Of course, again I could have saved typing labour, by replacing \global\def with \gdef (see above) and \global\edef with \xdef. So this is an example with all four kinds of def: \def, \edef, \gdef and \xdef.