A guide to understanding expandability: when to write protected functions and when not to

Question

I'm having difficulty understanding (and appreciating) the concept of expandability. I'm very murky about understanding when and how expandability impacts me in writing code for my documents.

I've read Why isn't everything expandable?. The answer was interesting and useful, but it didn't get at the heart of what I'm curious about. I've also perused a number of the answers to other questions involving expandability: of particular interest was this post.

In respondence to a recent question of mine, it was explained that document commands are protected and hence not expandable. Understanding this allowed me to write what I wanted and to get the effect I expected.

And in a comment to another question of mine, it was explained how one should use \cs_new_protected:Npn "when the function does unexpandable jobs such as setting token lists or sequences."

For years, I've been writing code like

\newcommand{\currentanswer}{}
\newcommand{\setcurrentanswer}[1]{\renewcommand{\currentanswer}{#1}}

knowing that after calling \setcurrentanswer, any call to \currentanswer will result in the desired output. Am I relying upon (un)expandability here? I'm not really sure; I only know that it does what I want. Then there are times I know I can throw in a \protect to get the result I want: but, I really don't understand the why of it; I just know it gets the job done.

Recently, I've been trying to learn some LaTeX3: the more I play with it, the more I like it. LaTeX---which I always thought was pretty powerful---is suddenly much more powerful and transparent in the manner that macros and functions can be defined. But now, I also seem to be running up against this issue of expandability, whereas before I could blithely go about my business ignorant of some of the subtlies of what I was doing.

While I am asking multiple questions here, I suspect that they really have much the same answer: hence I'm not splitting them across multiple posts.

1. Could someone take the time to explain some of the nuances of expandability, or, if not, point me to a good reference?

2. How do I know when I'm working with a protected function/macro?

3. Is protected and unexpandable the same thing?

4. Could someone explain the preference for protected functions in LaTeX3?

5. And finally, apart from the answers to the above questions, why would it be preferrable to protect functions which perform unexpandable tasks: such as setting tokens and sequences? (I am very interested in understanding this last question.)

Answer 1

An expandable command is one which can be converted 'fully' into its output inside a TeX \edef or \write (and a few other places). Thus for example

\def\testa{\testb}
\def\testb{\testc}
\def\testc{d}
\edef\teste{\testa}
\show\teste

will give

> \teste=macro:
->d.

i.e. all of the steps have been expanded, and we have just characters.

For text, this is nice and simple, but when you get TeX primitives involved things are more complex as some are expandable and some are not. Broadly, anything which performs an assignment is unexpandable. So if we have

\def\testa{\testb}
\def\testb{\testc}
\def\testc{\def\ARG{d}}
\def\ARG{}
\edef\teste{\testa}
\show\teste

we get

> \teste=macro:
->\def {d}.

Notice how the \def is left unchanged but the \ARG has vanished: it got expanded to what it is defined as (empty).

e-TeX allows us to define a protected macro. These do not expand inside an \edef, so

\def\testa{\testb}
\def\testb{\testc}
\def\testc{\def\NOTARG{d}}
\protected\def\NOTARG{}
\edef\teste{\testa}
\show\teste

now yields

> \teste=macro:
->\def \NOTARG {d}.

There is a subtle but important point here: \def is an unexpandable primitive, while \NOTARG is now a protected macro. You can tell that \NOTARG is protected using \show:

> \NOTARG=\protected macro:
->.

where the \protected tells us what we need to know. However, you have to know that \def is not expandable.

---

In the LaTeX3 documents, rather than expect people to learn the rules we've gone with a different approach: we document which functions are expandable (they are marked with a star). The reason everything else is then protected is that 'partial' expansion is a real issue. If you do

\def\testa{\let\testb\testc}
\edef\testb{\testa}

you get

! Undefined control sequence.
\testa ->\let \testb 
                     \testc 

as \let is unaffected by the \edef but \testb is undefined. This gets worse when you look at 'real' documents, as the problem can be hidden many layers down.

Many of the issues people see in real LaTeX2e documents, for example where they forget \protect and have trouble, would be bypassed if most commands were protected. In general, you find a lot more (La)TeX code that is not expandable than code that is, so the position for LaTeX3 is that this is the exception, certainly for document commands. (Typesetting is not expandable, and that's what happens in documents.)

This leads us on to what I call the 'sheep and goats' approach to protected functions: all LaTeX3 code is either protected or fully expandable ['safe' (will give the expected result) inside \edef/x-type expansion], even if we are talking about auxiliary functions. The result is that we can always be sure if a function can be used in an expansion context: if it can, it's marked with a star, otherwise it will be protected and won't expand part way. So the 'correct' way to write LaTeX3 code is that if you use anything that is not expandable (i.e. not starred in the documentation) in your code, then you have to use \cs_new_protected:Npn or similar, and not \cs_new:Npn, etc.

Answer 2

in

\newcommand{\currentanswer}{}
\newcommand{\setcurrentanswer}[1]{\renewcommand{\currentanswer}{#1}}

your setting command is not expandable, it expands to \renewcommand which itself expands to some other things but ultimately it makes an assignment (with \def) it sets \currentanswer to the tokens passed in as #1. However the macro \currentanswer ise defined to expand to the value passed in so is safe to use in an expansion context so long as the value passed in is safe.

Note as discussed in Advantages and disadvantages of fully expandable macros It doesn't really make sense to categorize commands as expandable or not, any macro by definition expands to its definition, the differences come from the context, whether you are only doing expansion or whether you are in TeX's normal mode of operation with expansion interleaved with assignments and other non-expandable primitive operations.

1. Can try, but see also the above link

2. You need to look at its documentation or its definition. There are two distinct protection mechanisms LaTeX \protect system and the e-tex \protected\def system.

   *\show\pounds
   > \pounds=macro:
   ->\protect \pounds  .
   <*> \show\pounds
   
   ? 
   
   *\protected\edef\foo{abc}
   
   *\show\foo
   > \foo=\protected macro:
   ->abc.
   <*> \show\foo
   
   ? 
   

   The above interactive session shows that \pounds is defined to be \protect\pounds (The second token there has a space at the end of its name) and so relies on LaTeX definition of \protect.

   \foo is defined using the e-TeX protection mechanism (which is what LaTeX3 commands use mostly) and this shows up in the \show output. The token \foo is protected by the tex engine itself (so will not expand in expansion only contexts) and does not need an internal macro to hold the real definition as is the case with \pounds.

3. No. Unexpandable primitives like \def work 'in the stomach', protected commands are expandable macros, but you need them not to expand in some contexts (for example so they write to auxiliary file as the command name rather than its current definition) A protected command is normally expandable but acts as an unexpandable command in an expansion only context.

4. They are preferred as they lead to less surprises. (Normally).

5. An unprotected command will fail if used in a write or an \edef so for example in \caption where you need to write the text to the .toc file.

Answer 3

The entire topic of expandability requires some more-or-less steep learning curve.

This is a lengthy answer. Coming down to the point: I am trying to summarize the relevant essence of Where do I start LaTeX programming? .

And given some good will, it requires knowledge of tools which show what is the actual outcome of your code.

The concept of expandability can be understood if you learn more about the involved internals, i.e. how to define macros without or with expanding the argument.

Putting it to the point: what is the outcome of \renewcommand{\currentanswer}{\content} ? Does \currentanswer expand to \content which, in turn, expands to whatever it "contains"? Or does \currentanswer contain a copy of "the value" of \content after the \renewcommand? The difference is, of course, only measurable if \content changes right after your \renewcommand.

Although this appears to be a technical distinction, it is directly related to your question.

What happens if you write \section{\currentanswer} if you modify the value of \currentanswer immediately after the \section command?

Another question is: what happens if you write \section{\tikz \draw ... ;} ? Clearly, this is a different use-case: we placed some executable code inside of the section argument.

The short answer is: you may need to learn the difference between \def and \edef ("expanded" \def). And you should know that \section{} uses \edef to collect its arguments; the same holds for other "movable" arguments (anything provided to sectioning commands, index, cross-references, etc) in LaTeX. The \renewcommand boils down to a \def; it does not expand the argument. Inside of the body of \edef, any conditionals will be expanded. Anything which is "executable" will not be expanded and needs to be protected. This is a rule-of-thumb; you would still need to learn what "executable" means. But typically, the intuition helps enough (like "a macro which contains a value is not expandable" whereas "\tikz ... ; certainly is").

Interesting helper utilities are \show<macroname> or \message{The meaning of content is \meaning\content^^J} to inspect what you wrote in your own code.

Furthermore, in LaTeX, any "executable" argument should be protected if you provide it as argument to some sectioning command. And protection typically means \protect\macro.

I suggest to read Where do I start LaTeX programming? if you want to learn more about these concepts in general. It will probably help you to understand the related high-level concepts of LaTeX3.

Answer 4

Expansion of TeX macros is from outside to inside. Therefore it is complicated to correctly write nesting macros in all the time.

With my functional package, all functions are protected, but the evaluation of composite functions is from inside to outside, just like Lua language. This would make programming with TeX much easier.

\documentclass{article}
\usepackage{functional}
\begin{document}

\IgnoreSpacesOn
\PrgNewFunction \MathSquare { m } {
  \IntSet \lTmpaInt { \IntEval {#1 * #1} }
  \Result { \Value \lTmpaInt }
}
\IgnoreSpacesOff
\MathSquare{5}
\MathSquare{\MathSquare{5}}

\end{document}

Note that with this package, you need to pass return values of functions with \Result command.

when to write protected functions and when not to

In my opinion, ALL user-defined functions should be protected and only their return values (as tl variables) could be expandable. And this package would help users to fulfil it.

\TlSet \lTmpaTl {\MathSquare{5}}
\TlLog {\Value \lTmpaTl}

In the above example, \MathSquare function is protected, and its return value is stored in \lTmpaTl variable, which could be expandable.

Answer 5

Some additional note on how to write expl3 code regarding expandability.

---

Recall that

• a function is called "expandable" if it works as you desire in expansion-only context.

• within the unexpandable ones, the ones that

  • does not error out in expansion-only context, and
  • does the right thing when it's eventually executed in normal context

  are called robust. The rest are called "fragile".

• you should protect the unexpandable functions because otherwise it may be fragile, and fragile commands does the wrong thing when you accidentally/need to put it in expansion-only context (e.g. moving argument).

---

Imagine you want to do something like (pseudocode in C-like syntax)

result = f(g(h(argument)))

where argument and result are variables, f, g, h are functions that should return some result.

If all of them are expandable, you can simply do

\tl_set:Ne \result { \f:e { \g:e { \h:e { \argument } } } }

However, what if some are not expandable? Obviously you can't just pass it inside the argument, there's no way for the outer macro to access the result.

The common TeX idiom here is that the function stores the result into somewhere.

In expl3 code, the destination is usually provided as a N-type argument. (compare with pgf, math result are stored in \pgfmathresult)

Assume f is not expandable, it would normally have the form

\f:Nn <target> <argument>

or

\f:N <data>   % for example regex_replace_all

In such cases you would need to "unroll" the call chain:

\h:Ne \result {\argument}
\g:Ne \result {\result}
\f:Ne \result {\result}