# Expanding a parameter token argument to \g@addto@macro

Consider the following example.

By default \g@addto@macro doesn't expand its arguments, which can have unexpected results when passing a macro as an argument, if the macro value is changed later.

To expand the macro argument, I used the workaround shown below. Does this approach have any disadvantages? Is there a better way to do this?

In this example code, \append doesn't expand its argument. \newappend does. They both append the argument #1 (followed by a comma) to the macros \slist and \newslist respectively.

\documentclass[12pt]{article}

\def\slist{}
\def\newslist{}

\makeatletter
\def\append#1
{
}
\def\newappend#1
{
\def\tmpmacroval{#1}
\next
}
\makeatother

\def\foo{fooval}

\begin{document}
\append{\foo}

\newappend{\foo}

\def\foo{someotherval}

\verb|\slist| is \slist

\verb|\newslist| is \newslist
\end{document}


ADDENDUM: I omitted the context of this question for simplicity, but the argument to \append/\newappend is basically a label. This is normally just a string of characters, but I started using a variant which is a single macro which expands to a string, with a string appended. E.g. \theletternum:2019.09.30, where letternum is a counter, defined by \stepcounter{letternum}. In this case, the argument needs to be expanded before being appended to \slist/\newslist. I don't currently think that the input will get more complicated than that, but of course it's hard to see the future.

• Well you are fully expanding your \tmpmacroval/\foo command. This could explode if the command contains stuff that better shouldn't be passed through an \edef. Better use \protected@edef, or expand only once. And consider to use expl3 which has readymade commands for this. Feb 14, 2020 at 17:35
• expl3 has readymade commands for what? Feb 14, 2020 at 17:43
• To append content to commands and to expand them if wanted. Feb 14, 2020 at 17:46
• @UlrikeFischer Oh, I see. Can you provide a relevant link? Something like tex.stackexchange.com/a/67372/3406 perhaps? Feb 14, 2020 at 17:48
• most latex commands will fail in an edef Feb 14, 2020 at 18:03

First off: as I've written several times, TeX is not free-form. There's a huge difference between

\def\append#1
{
}


and the code you probably want, namely

\def\append#1{%
}


With your code the argument to \append is determined as whatever comes after it up to the first space token. So if you write

\append{a}Some words


the result is that {a}Some is appended to the list.

Now to your problem. It's not really clear what kind of input you expect. If the input is either text or a single macro that expands to text, you can simply do

\expandafter\append\expandafter{\foo}


If you want instead to pass arbitrary text that may contain macros to be expanded, then you need great care. If this text contains something like \ref and the object referenced is still unknown, then you're basically doomed even if you use \protected@edef. The code

\documentclass{article}

\makeatletter
\def\slist{}
\def\append#1{%
\protected@xdef\slist{\unexpanded\expandafter{\slist}#1,}%
}
\makeatother

\begin{document}

\append{\ref{what}}

\show\slist

\end{document}


will print on the terminal

> \slist=macro:
->\protect \G@refundefinedtrue {\protect \mbox  {\protect \normalfont  \protect
\bfseries  ??}}\protect \GenericWarning  {               }{LaTeX Warning: Refe
rence what' on page 1 undefined},.


which is probably something you don't really like to see.

Therefore I'll assume the argument to \append is either text or a single macro.

\documentclass{article}

\makeatletter
\def\slist{}
\def\append{\@ifstar{\append@expand}{\append@text}}
\def\append@expand#1{\expandafter\append@text\expandafter{#1}}%
\makeatother

\newcommand{\foo}{something}

\begin{document}

\append{a}\append*{\foo}

\show\slist

\end{document}


This will print the expected

> \slist=macro:
->a,something,.


I'd be wary of suggesting the simpler

\documentclass{article}

\makeatletter
\def\slist{}
\def\append#1{%
}
\makeatother

\newcommand{\foo}{something}

\begin{document}

\append{a}\append*{\foo}

\show\slist

\end{document}


that will do the same. You can use this simpler approach if your input is always a string of characters or a single macro expanding to a string of characters (‘string’ in the nontechnical sense, meaning ‘sequence’).

• Hi @egreg. To answer your implied question, the input is basically a label. This is normally just a string of characters, but I've also used a variant which is a single macro which expands to a string, with a string appended. E.g. \theletternum:2019.09.30, where letternum is a counter, defined by \stepcounter{letternum}. And of course this latter variant blows up if it's not expanded first. I don't currently think that the input will get more complicated than that, but of course it's hard to see the future. Which version would you recommend for this use case? Feb 16, 2020 at 12:02
• Can you explain what your \append version containing \protected@xdef\slist{\unexpanded\expandafter{\slist}#1,}% does? I assume the purpose of this example is to show that bad things can happen if you pass in an arbitrary macro. Feb 16, 2020 at 12:39
• @FaheemMitha If your input is guaranteed to be fully expandable, as in the case you show, you can use \xdef\slist{\slis}#1,} Feb 16, 2020 at 16:17

As @UlrikeFischer and @DavidCarlisle pointed out, using \edef with arbitrary material is likely to cause errors. For instance, this will be the case with material containing \textbf, and there are plenty of other cases.

# Two ways in LaTeX2e style

I present two techniques in LaTeX2e style that use \protected@edef so that macros defined with \DeclareRobustCommand are not affected. This way, the added material may contain \textbf.

These techniques also don't overwrite the \next macro, which is always nice (well, they do but automatically restore it when finished). Also note that your \tmpmacroval macro wasn't really useful.

Note that your list is duly modified despite the grouping I added, because \g@addto@macro uses \xdef, which performs a global assignment.

## First technique

\documentclass{article}

\newcommand{\newslist}{}

\makeatletter
\newcommand{\newappend}[1]{%
\begingroup
\next
}
\makeatother

\newcommand{\foo}{\textbf{foo}val}

\begin{document}

\newappend{\foo}
\renewcommand{\foo}{someotherval}
\verb|\newslist| is \newslist.

\end{document}


Explanation

Let's assume that TeX has just expanded \newappend{〈argument〉}. This leaves the following at the front of the input stream:

\begingroup
\next


Upon execution of \begingroup, TeX opens a new group. Then he sees the line:

\protected@edef\next{\endgroup\noexpand\g@addto@macro\noexpand\newslist{〈argument〉,}}


After \protected@edef has been expanded and all the resulting tokens processed, the macro \next is defined in the current group. The definition is local to this group, and the previous definition of \next (possibly: none) will be automatically restored by TeX as soon as this group ends—that is the main purpose of TeX groups. Now, the next token in the input stream is the \next control sequence token. Since it is a macro, TeX expands it, which causes this \next token to be replaced by:

\endgroup\g@addto@macro\newslist{〈expanded argument〉,}


(the \noexpand tokens have been removed as part of the execution of an \edef in the implementation of \protected@edef). The next token in the input stream is thus \endgroup now. TeX executes it (it is unexpandable, thus it is handled in TeX's stomach, contrary to macros), which closes the current group. As said, this has the effect of cancelling the (re-)definition of \next that we did. This is clean. What remains at the front of the input stream is now

\g@addto@macro\newslist{〈expanded argument〉,}


which is exactly what you wanted.

## Second technique

\documentclass{article}

\newcommand{\newslist}{}

\makeatletter
\newcommand{\newappend}[1]{%
\begingroup
\expandafter
\endgroup
\next
}
\makeatother

\newcommand{\foo}{\textbf{foo}val}

\begin{document}

\newappend{\foo}
\renewcommand{\foo}{someotherval}
\verb|\newslist| is \newslist.

\end{document}


Same output as in the previous example.

# Examples with expl3

expl3 has many facilities to deal with list-like material. I'll show some examples with clist, i.e. comma-separated lists. This allows easy direct setting from user input, since commas can be entered in the text. For more general lists, you can use seq variables. For lists of (key,value) pairs, there is the prop module (“property lists”). All these modules are built in the expl3 language, thus available after \usepackage{expl3} or \usepackage{xparse}, which loads expl3. There is no need to load anything else.

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\clist_new:N \g_faheem_clist

\cs_new_protected:Npn \faheem_append:Nn #1#2
{
\clist_gput_right:Nn #1 {#2} % 'g' for “global”
}

% Variant that performs one expansion step on the second argument
\cs_generate_variant:Nn \faheem_append:Nn { No }
% Variant that performs \edef-like expansion on the second argument
\cs_generate_variant:Nn \faheem_append:Nn { Nx } % \edef-like expansion
% There are also 'e' to perform \expanded-like expansion, 'f' for
% \romannumeral-like expansion, etc.

\NewDocumentCommand \newList { m }
{
\clist_new:N #1
}

% In fact, for serious work, better do:
%
% \cs_new_protected:Npn \faheem_new_list:N #1
%   {
%     \clist_new:N #1
%   }
%
% and make \newList call \faheem_new_list:N or \faheem_new_list:c as I do for
% the \append* functions below (the 'c' variant would allow you to choose
% arbitrary names for the clist variable: you would pass whatever name you
% want to the 'c' variant, normally without a leading slash, otherwise that
% would create a strange beast: \\the_passed_name).

\NewDocumentCommand \clearList { O{\g_faheem_clist} }
{
\clist_gclear:N #1          % 'g' for “global”
}

\NewDocumentCommand \appendNoExp { O{\g_faheem_clist} m }
{
\faheem_append:Nn #1 {#2}
}

\NewDocumentCommand \appendOneExp { O{\g_faheem_clist} m }
{
\faheem_append:No #1 {#2}
}

\NewDocumentCommand \appendFullExp { O{\g_faheem_clist} m }
{
\faheem_append:Nx #1 {#2}
}

% This requires a recent LaTeX release (2020 or later?)
\NewDocumentCommand \appendTextExpand { O{\g_faheem_clist} m }
{
% \text_expand:n works inside an 'x'-type argument because it is fully
% expandable, as indicated by the star in the margin in interface3.pdf.
\faheem_append:Nx #1 { \text_expand:n {#2} }
}

\NewDocumentCommand \useList { O{\g_faheem_clist} m }
{
\clist_use:Nn #1 {#2}
}
\ExplSyntaxOff

\newcommand*{\showres}[1]{\makebox[0pt][r]{#1.~}\ignorespaces}

\begin{document}

\newList{\somelist}
\appendNoExp{first in default list}
\appendNoExp[\somelist]{first in somelist}

\def\foobar{element with \textbf{non-expandable}\_material}
\def\baz{quux}
\appendOneExp{\foobar}
\appendOneExp[\somelist]{\baz}

\showres{1}
Default list: \useList{, }\par
\verb|\somelist|: \useList[\somelist]{, }

\def\fooleveli{\foolevelii}
\def\foolevelii{\fooleveliii}
\def\fooleveliii{other element}
\appendFullExp{\fooleveli}

\showres{2}
Default list: \useList{, }

\renewcommand{\fooleveliii}{\textbf{non-expandable} \emph{stuff}}
% Expand again \fooleveli, \foolevelii and (the new) \fooleveliii, this time
% with a method that won't break with LaTeX non-expandable commands defined
% with \DeclareRobustCommand.
\appendTextExpand{\fooleveli}

\showres{3}
Default list: \useList{, }.

\def\foobar{overwrite}
\def\fooleveli{overwrite}

\showres{4}
Default list: \useList{, }.\par
\verb|\somelist|: \useList[\somelist]{, }.

\showres{5}
With X as separator: \useList{X}.

\clearList                      % clear the default list

\showres{6}
Y\useList{X}Y

\end{document}


Note: for expanding text material without causing errors due to non-expandable, non-engine-protected commands like \textbf, recent expl3 has \text_expand:n, which is in the spirit of \protected@edef (see Joseph Wright's blog and interface3.pdf).

• I added an example using \text_expand:n. Feb 15, 2020 at 10:02
• Hi @frougon. Trying to understand your examples, but to start with I don't understand your usage of \begingroup and \endgroup in the first example. Can you expand on that? Feb 15, 2020 at 16:28
• Hi @FaheemMitha, please check the updated answer. Feb 15, 2020 at 16:54
• Hi @frougon. Thanks for the explanation. So once \next is expanded, the definition of \next disappears? I see. That's clever. I suppose the idea is that one should avoid "treading" on any pre-existing definition of \next? Feb 15, 2020 at 17:02
• Yes, that's the idea. To be accurate, the definition is cancelled when the group ends (i.e., when the \endgroup from our macro is executed), which is a bit later than “once \next is expanded”. But just a tiny bit. Once all tokens from the replacement text of \newappend have been digested, you can be sure that \next has been restored (possibly as undefined), which is what matters in practice. Feb 15, 2020 at 17:04
1. With your code a lot of unwanted space tokens come into being:
Due to \endlinechar a return-character (of category code 5(end of line)) is added at the right end of every line. After producing character-tokens like {1(begin group) and }2(end group) and 112(other) (as in #1=#6(parameter)112(other)) the reading apparatus of (La)TeX is in state m (middle of line) and therefore both following category code 5 return characters and following space characters will get tokenized as space tokens. Space tokens in turn yield visible horizontal glue in (restricted) horizontal mode.

2. \edef instead of \protected@edef might expand things that should not be expanded further.

3. In case the argument of \newappend contains hashes (#), two consecutive hashes will collapse into a single hash when \tmpmacroval is expanded. The same happens when \next gets expanded. \g@addto@macro in turn doubles hashes. In other words: \newappend divides amounts of consecutive hashes by 2.

Rectifying 1 and 2 could turn your code into something like this:

\documentclass[12pt]{article}

\def\slist{}
\def\newslist{}

\makeatletter
\def\newappend#1{%%%%
\def\tmpmacroval{#1}%%%%
\next
}%%%
\makeatother

\def\foo{fooval}

\begin{document}
\append{\foo}

\newappend{\foo}

\def\foo{someotherval}

\verb|\slist| is \texttt{\meaning\slist}

\verb|\newslist| is \texttt{\meaning\newslist}

% Here are sequences with four consecutive hashes:
\def\foo{\def\noexpand\bar####1{Arg of \string\bar\space is ####1}}

\append{\foo}

\newappend{\foo}

\verb|\slist| is \texttt{\meaning\slist}

\verb|\newslist| is \texttt{\meaning\newslist}

\end{document}


When you look at the last line of the resulting output, you notice that the amount of four consecutive hashes #### in the source gets halved and therefore \newslist contains sequences of two hashes only. (When expanding \newslist, amounts of hashes get halved again, thus expandiing \newslist yields the tokens:
fooval,\def\bar#1{Arg of \bar is #1}, ; the second \bar is stringified.)

With TeX-engines where both the \expanded-primitive and the \unexpanded-primitive is available, you can also rectify issue 3 (the collapsing of two consecutive hashes into a single hash during expansion). I'd probably do something like this:

\documentclass{article}

\newcommand\exchange[2]{#2#1}

\makeatletter
% The following three lines form the \protected@edef-mechanism:
\let\@@protect\protect
\let\protect\@unexpandable@protect
\afterassignment\restore@protect
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\xdef#1{%
% \unexpanded inside \edef yields doubling of amounts of hashes.
\unexpanded\expandafter{%
\romannumeral0%
% \romannumeral0-\exchange-trickery to get the things that are to be appended
% expanded fully via \expanded, and to get the macro where things are to be
% appended expanded once; during expansion - no matter if expansion
% is only once or fully - amounts of hashes get halved:
\expandafter\exchange\expandafter{\expanded{#2}}%
{\exchange{ }\expandafter#1}%
}%
}%
}%
\makeatother

\newcommand\slist{}
\newcommand\newslist{}

\newcommand\foo{fooval}

\begin{document}
\append{\foo}

\newappend{\foo}

\def\foo{someotherval}

\verb|\slist| is \texttt{\meaning\slist}

\verb|\newslist| is \texttt{\meaning\newslist}

% Here are sequences with two consecutive hashes:
\def\foo{\def\noexpand\bar##1{Arg of \string\bar\space is ##1}}

\append{\foo}

\newappend{\foo}

\verb|\slist| is \texttt{\meaning\slist}

\verb|\newslist| is \texttt{\meaning\newslist}

\end{document}


When you look at the last line of the resulting output, you notice that summa summarum the amount of two consecutive hashes ## in the source does not get halved and therefore \newslist` contains sequences with the same amount of hashes.