# A guide to understanding expandability: when to write protected functions and when not to

I'm having difficulty understanding (and appreciating) the concept of expandability. I'm very murky about understanding when and how expandability impacts me in writing code for my documents.

I've read Why isn't everything expandable?. The answer was interesting and useful, but it didn't get at the heart of what I'm curious about. I've also perused a number of the answers to other questions involving expandability: of particular interest was this post.

In respondence to a recent question of mine, it was explained that document commands are protected and hence not expandable. Understanding this allowed me to write what I wanted and to get the effect I expected.

And in a comment to another question of mine, it was explained how one should use \cs_new_protected:Npn "when the function does unexpandable jobs such as setting token lists or sequences."

For years, I've been writing code like

\newcommand{\currentanswer}{}


knowing that after calling \setcurrentanswer, any call to \currentanswer will result in the desired output. Am I relying upon (un)expandability here? I'm not really sure; I only know that it does what I want. Then there are times I know I can throw in a \protect to get the result I want: but, I really don't understand the why of it; I just know it gets the job done.

Recently, I've been trying to learn some LaTeX3: the more I play with it, the more I like it. LaTeX---which I always thought was pretty powerful---is suddenly much more powerful and transparent in the manner that macros and functions can be defined. But now, I also seem to be running up against this issue of expandability, whereas before I could blithely go about my business ignorant of some of the subtlies of what I was doing.

While I am asking multiple questions here, I suspect that they really have much the same answer: hence I'm not splitting them across multiple posts.

1. Could someone take the time to explain some of the nuances of expandability, or, if not, point me to a good reference?

2. How do I know when I'm working with a protected function/macro?

3. Is protected and unexpandable the same thing?

4. Could someone explain the preference for protected functions in LaTeX3?

5. And finally, apart from the answers to the above questions, why would it be preferrable to protect functions which perform unexpandable tasks: such as setting tokens and sequences? (I am very interested in understanding this last question.)

-

An expandable command is one which can be converted 'fully' into it's output inside a TeX \edef or \write (and a few other places). Thus for example

\def\testa{\testb}
\def\testb{\testc}
\def\testc{d}
\edef\teste{\testa}
\show\teste


will give

> \teste=macro:
->d.


i.e. all of the steps have been expanded, and we have just characters.

For text, this is nice and simple, but when you get TeX primitives involved things are more complex as some are expandable and some are not. Broadly, anything which performs an assignment is unexpandable. So if we have

\def\testa{\testb}
\def\testb{\testc}
\def\testc{\def\ARG{d}}
\def\ARG{}
\edef\teste{\testa}
\show\teste


we get

> \teste=macro:
->\def {d}.


Notice how the \def is left unchanged but the \ARG has vanished: it got expanded to what it is defined as (empty).

e-TeX allows us to define a protected macro. These do not expand inside an \edef, so

\def\testa{\testb}
\def\testb{\testc}
\def\testc{\def\NOTARG{d}}
\protected\def\NOTARG{}
\edef\teste{\testa}
\show\teste


now yields

> \teste=macro:
->\def \NOTARG {d}.


There is a subtle but important point here: \def is an unexpandable primitive, while \NOTARG is now a protected macro. You can tell that \NOTARG is protected using \show:

> \NOTARG=\protected macro:
->.


where the \protected tells us what we need to know. However, you have to know that \def is not expandable.

In the LaTeX3 documents, rather than expect people to learn the rules we've gone with a different approach: we document which functions are expandable (they are marked with a star). The reason everything else is then protected is that 'partial' expansion is a real issue. If you do

\def\testa{\let\testb\testc}
\edef\testb{\testa}


you get

! Undefined control sequence.
\testa ->\let \testb
\testc


as \let is unaffected by the \edef but \testb is undefined. This gets worse when you look at 'real' documents, as the problem can be hidden many layers down.

Many of the issues people see in real LaTeX2e documents, for example where they forget \protect and have trouble, would be bypassed if most commands were protected. In general, you find a lot more (La)TeX code that is not expandable than code that is, so the position for LaTeX3 is that this is the exception, certainly for document commands. (Typesetting is not expandable, and that's what happens in documents.)

This leads us on to what I call the 'sheep and goats' approach to protected functions: all LaTeX3 code is either protected or fully expandable ['safe' (will give the expected result) inside \edef/x-type expansion], even if we are talking about auxiliary functions. The result is that we can always be sure if a function can be used in an expansion context: if it can, it's marked with a star, otherwise it will be protected and won't expand part way. So the 'correct' way to write LaTeX3 code is that if you use anything that is not expandable (i.e. not starred in the documentation) in your code, then you have to use \cs_new_protected:Npn or similar, and not \cs_new:Npn, etc.

-
Can you pls expand on "...all LaTeX3 code is either protected or expandable", seems obvious and the same can be said for LaTeX2e. – Yiannis Lazarides Jan 27 '13 at 18:11
Thanks for the explanation. – Yiannis Lazarides Jan 27 '13 at 18:56
@JosephWright: That is the same approach in ConTeXt MkIV. Any code that cannot be used inside an \edef is explicitly defined as unexpandable (using \unexpanded\def... where \unexpanded is ConTeXt's alias for \protected). – Aditya Jan 27 '13 at 19:31
@Aditya Thanks for that: I'd not read the ConTeXt code in enough detail to pick that up. Obviously we've all come to the same conclusions :-) – Joseph Wright Jan 27 '13 at 20:21

in

\newcommand{\currentanswer}{}


your setting command is not expandable, it expands to \renewcommand which itself expands to some other things but ultimately it makes an assignment (with \def) it sets \currentanswer to the tokens passed in as #1. However the macro \currentanswer ise defined to expand to the value passed in so is safe to use in an expansion context so long as the value passed in is safe.

Note as discussed in Advantages and disadvantages of fully expandable macros It doesn't really make sense to categorize commands as expandable or not, any macro by definition expands to its definition, the differences come from the context, whether you are only doing expansion or whether you are in TeX's normal mode of operation with expansion interleaved with assignments and other non-expandable primitive operations.

2. You need to look at its documentation or its definition. There are two distinct protection mechanisms LaTeX \protect system and the e-tex \protected\def system.

*\show\pounds
> \pounds=macro:
->\protect \pounds  .
<*> \show\pounds

?

*\protected\edef\foo{abc}

*\show\foo
> \foo=\protected macro:
->abc.
<*> \show\foo

?


The above interactive session shows that \pounds is defined to be \protect\pounds (The second token there has a space at the end of its name) and so relies on LaTeX definition of \protect.

\foo is defined using the e-TeX protection mechanism (which is what LaTeX3 commands use mostly) and this shows up in the \show output. The token \foo is protected by the tex engine itself (so will not expand in expansion only contexts) and does not need an internal macro to hold the real definition as is the case with \pounds.

3. No. Unexpandable primitives like \def work 'in the stomach', protected commands are expandable macros, but you need them not to expand in some contexts (for example so they write to auxiliary file as the command name rather than its current definition) A protected command is normally expandable but acts as an unexpandable command in an expansion only context.

4. They are preferred as they lead to less surprises. (Normally).

5. An unprotected command will fail if used in a write or an \edef so for example in \caption where you need to write the text to the .toc file.

-
LaTeX3 code does not use \protect at all, so it's not 'mostly' e-TeX, it's all e-TeX :-) – Joseph Wright Jan 27 '13 at 17:58
@JosephWright yes I meant that the latex3 commands are mostly protected but you could defined unprotected ones when needed. – David Carlisle Jan 27 '13 at 18:10
Thank you for directing to your other post. Very enlightening. – A.Ellett Jan 27 '13 at 23:39

The entire topic of expandability requires some more-or-less steep learning curve.

This is a lengthy answer. Coming down to the point: I am trying to summarize the relevant essence of Where do I start LaTeX programming? .

And given some good will, it requires knowledge of tools which show what is the actual outcome of your code.

The concept of expandability can be understood if you learn more about the involved internals, i.e. how to define macros without or with expanding the argument.


Although this appears to be a technical distinction, it is directly related to your question.

What happens if you write \section{\currentanswer} if you modify the value of \currentanswer immediately after the \section command?

Another question is: what happens if you write \section{\tikz \draw ... ;} ? Clearly, this is a different use-case: we placed some executable code inside of the section argument.

The short answer is: you may need to learn the difference between \def and \edef ("expanded" \def). And you should know that \section{} uses \edef to collect its arguments; the same holds for other "movable" arguments (anything provided to sectioning commands, index, cross-references, etc) in LaTeX. The \renewcommand boils down to a \def; it does not expand the argument. Inside of the body of \edef, any conditionals will be expanded. Anything which is "executable" will not be expanded and needs to be protected. This is a rule-of-thumb; you would still need to learn what "executable" means. But typically, the intuition helps enough (like "a macro which contains a value is not expandable" whereas "\tikz ... ; certainly is").

Interesting helper utilities are \show<macroname> or \message{The meaning of content is \meaning\content^^J} to inspect what you wrote in your own code.

Furthermore, in LaTeX, any "executable" argument should be protected if you provide it as argument to some sectioning command. And protection typically means \protect\macro.

I suggest to read Where do I start LaTeX programming? if you want to learn more about these concepts in general. It will probably help you to understand the related high-level concepts of LaTeX3.

-
The link you provided was very helpful. Thanks – A.Ellett Jan 27 '13 at 23:42