# Expandable full expansion of tokens that preserves catcodes

Is it possible to fully expand tokens in an expandable manner and preserves category codes? I'd like to do this just using pdfTeX.

I'm looking for something that would work like this:

\def\foo{foo}
\fullyexpand{\foo bar baz}


which would expand to the 10 tokens "foobar baz" and retain the category codes.

If we drop the expandable requirement, then this is easy.

\def\fullyexpand#1{\edef\fetemp{#1}\fetemp}


If we drop the retain catcode requirement, it's doable, but less straight-forward. Here's the best I could come up with.

\def\gobbleprefix#1#2\femarker{%
\ifnum\escapechar<0
#1%
\else\ifnum\escapechar>255
#1%
\fi\fi
#2%
}
\def\fullyexpand#1{%
\expandafter\expandafter\expandafter\gobbleprefix
\expandafter\string\csname#1\endcsname\femarker
}


I was hoping to combine the ε-TeX extension \scantokens with the above (by an appropriate, trivial modification of \gobbleprefix), but of course, that does not work. \scantokens's expansion is empty and pdfTeX just acts like it has opened a new file.

(As an aside, it amuses me that not expanding tokens is expandable—using \unexpanded—but expanding tokens isn't, at least not obviously so.)

-
I will write an answer to this question today or tomorrow. In the meantime, you might be interested in \expandsome{\foo{ } \bar{\expandthis\foo}\expandthis\bar{abc}} which will expand precisely the tokens following \expandthis, in an expandable manner. The code can be found online, or at this question. --- Actually, it would help me if someone has a list of all primitives that should be expanded. –  Bruno Le Floch Feb 15 '11 at 9:32

Did you try using \romannumeral? This is used a lot for this type of thing (see for example the \exp_args:Nf concept in expl3):

\def\fullyexpand#1{\romannumeral - 0#1}


This works because TeX will keep expanding #1 looking for a number, which will always turn out to be negative, so the Roman numeral will vanish. Note that this solution will stop on the first non-expandable token, unlike an \edef which will keep going.

It's possible to build a function which can expand using \romannumeral 'around' unexpandable tokens. For example, the following code will work reasonably well:

\long\def\fullyexpand#1{%
\csname donothing\fullyexpandauxi{#1}{}%
}
\long\def\fullyexpandauxi#1{%
\expandafter\fullyexpandauxii\romannumeral -0#1\fullyexpandend
}
\long\def\fullyexpandauxii#1#2\fullyexpandend#3{%
\ifx\donothing#2\donothing
\expandafter\fullyexpandend
\else
\expandafter\fullyexpandloop
\fi
{#1}{#2}{#3}%
}
\long\def\fullyexpandend#1#2#3{\endcsname#3#1}
\long\def\fullyexpandloop#1#2#3{%
\fullyexpandauxi{#2}{#3#1}%
}
\def\donothing{}


However, this is not the same as \expanded, for a few reasons. First, my implementation will strip out spaces in the argument (as it does a loop, and TeX will skip spaces). Braces will also get stripped out. A bit of testing also reveals that \romannumeral will expand \protected functions here, whereas \expanded does not. I'd also note that the above code needs some guards adding for a blank (empty or all space) argument, as currently things fail in these cases.

You might be interested to see that there is code for an \expanded primitive in the repository for pdfTeX 1.50. I've no idea whether we'll ever see this released, but it would be handy in this situation.

On the aside, it is possible to use \scantokens expandably, but as you may have found this can be tricky and it is usually necessary to have a (non-expandable) change of \everyeof first. LuaTeX addresses this issue with the \scantextokens primitive, which combines this end-of-file stuff directly into the primitive. Of course, if you are using LuaTeX then the original problem is solvable anyway, since \expanded is available.

-
Wouldn't it also strip out leading numbers? –  TH. Dec 21 '10 at 18:11
@TH.: no, because -0 is already a complete number, only that is stripped. –  Philipp Dec 21 '10 at 18:16
@Joseph Wright: Exhaustive expansion in LuaTeX doesn't need \directlua because even current versions of LuaTeX have \expanded. –  Philipp Dec 21 '10 at 18:25
@Philipp: Oh, I see. I didn't even notice the backtick. It's interesting that TeX allows an optional space after numbers of that form. I guess it is expanding looking for the space, but that is very strange. Yeah, I guess it is: \romannumeral 5\space 3 produces v3. I'll bet this can be leveraged to expand everything. –  TH. Dec 21 '10 at 18:52
@Philipp @Joseph For Roman numerals TeX defines an empty expansion, if the number is zero or negative why do you need the minus? –  Yiannis Lazarides Dec 21 '10 at 18:56

After learning many tricks on this forum, I propose the following solution. I think it does what @TH. wanted, namely it expands everything, and when it meets a non-expandable token, it just stores it and continues. It uses a future package of mine, whose code can currently be found online, or as my answer to this question. This code is input in the first line below. Set whichever filename you want.

Note that I have not been careful to reset catcodes to their default value in ULcase.sty, so here, we don't need \catcode\:=11\catcode_=11\relax. One day, I'll clean this up.

% Code based on the extended Upper- and Lower-casing code found
% in the ULcase package.
\input ULcase.sty\relax

% ============ Table |fullyexpand|, |\fullyexpand|
% The |fullexpand| table expands every token that is expandable
% according to the test \FE_token_if_expandable:NTF.
%
% Then |\fullyexpand| is basically changing the case using a special
% "case table", |ULfullyexpand|. Here, |\MEA_trigger:f| is |\romannumeral|
% in disguise, forcing the full expansion of |\UL_to_case_aux:nn|
% (This is necessary for technical reasons.)
%
\def\fullyexpand{\MEA_trigger:f\UL_to_case_aux:nn{ULfullyexpand}}

% A few tests, building up to \FE_token_if_expandable:NTF.
%
\long\gdef\FE_token_if_expandable:NTF#1{%
\FE_token_if_defined:NTF#1%
{%
\FE_token_if_eq_noexpand_self:NTF#1%
{\use_ii:nn}%
{\FE_token_if_protected:NTF#1{\use_ii:nn}{\use_i:nn}}%
}%
{\use_ii:nn}%
}
\long\gdef\FE_token_if_defined:NTF#1{%
\ifdefined #1%
\expandafter\use_i:nn%
\else%
\expandafter\use_ii:nn%
\fi%
}%
\long\gdef\FE_token_if_eq_noexpand_self:NTF#1{%
\expandafter\ifx\noexpand#1#1%
\expandafter\use_i:nn%
\else%
\expandafter\use_ii:nn%
\fi%
}
% |\expandsome| only expands the tokens following |\expandthis|.
\expandsome{%
\long\gdef\FE_token_if_protected:NTF#1{%
\expandafter\FE_token_if_protected_aux:w\meaning#1%
\expandthis\string\protected\q_stop}%
}
% |\expandafter:nw{...}\foo| expands |\foo| before |...|.
\expandafter:nw{\long\gdef\FE_token_if_protected_aux:w#1}%
\string\protected#2\q_stop{%
\UL_if_empty:nTF{#1}%
}%

% ===== Building the table.
% We copy the standard definitions (in particular for braces)
% NB: maybe problem with \NoCaseChange.
\UL_new_table:n{ULfullyexpand}
%
% Spaces are just kept:
\UL_setup:nnn{ULfullyexpand}{ }{ }
%
% The default action is to check if #3 (the next token) is expandable.
% If it is, we expand it. Otherwise, we output it. "#2{#3}" is responsible
% for continuing the loop.
\long\gdef\UL_table_ULfullyexpand_default#1#2#3{%
\FE_token_if_expandable:NTF#1{%
\expandafter:nw{#2{#3}}#1%
}{%
\UL_to_case_output:n{#1}#2{#3}%
}%
}%
%
% Define |\noexpand|
\long\expandafter\gdef\csname UL_table_ULfullyexpand_%
\detokenize{\noexpand}\endcsname#1#2#3{%
\UL_to_case_output:n{#3}#1{#2}}
%
% Define |\detokenize|
\long\expandafter\gdef\csname UL_table_ULfullyexpand_%
\detokenize{\detokenize}\endcsname#1#2{%
\expandafter:nw{\FE_detok_unexp_aux:nnNn{#1}{#2}\detokenize}%
\romannumeral-\0}
%
% Define |\unexpanded|
\long\expandafter\gdef\csname UL_table_ULfullyexpand_%
\detokenize{\unexpanded}\endcsname#1#2{%
\expandafter:nw{\FE_detok_unexp_aux:nnNn{#1}{#2}\unexpanded}%
\romannumeral-\0}
%
% A helper for |\detokenize| and |\unexpanded|.
\long\gdef\FE_detok_unexp_aux:nnNn#1#2#3#4{%
\expandafter\UL_to_case_output:n\expandafter{#3{#4}}%
#1{#2}}%

% ===== Tests
\long\gdef\fooA#1{\fooC{\noexpand\fooA got #1} \fooB{a nice #1}}
\long\gdef\fooB#1{fooB\space got <#1>}
\protected\def\fooC#1{Not expanded.}
\long\gdef\a{\fullyexpand{Text: \fooA{argument}}}
\expandonce\a\expandonce\a
\long\gdef\b{Text: \fooC {\fooA got argument} fooB got <a nice argument>}
\checkoutput

\long\gdef\a{\fullyexpand{Text:%
\detokenize\expandafter{\fooA{argument}\fooA{hi}}}}
\long\xdef\b{Text:\detokenize{\fooC {\noexpand \fooA got argument} %
\fooB {a nice argument}\fooA {hi}}}
\expandonce\a\expandonce\a
\checkoutput

\long\gdef\a{\fullyexpand{Text:%
\unexpanded\expandafter{\fooA{argument}\fooA{hi}}}}
\long\xdef\b{Text:\unexpanded{\fooC {\noexpand \fooA got argument} %
\fooB {a nice argument}\fooA {hi}}}
\expandonce\a\expandonce\a
\checkoutput

\long\gdef\foo{foo}
\long\gdef\a{\fullyexpand{a\foo \ifnum1=1\number2 yes\else no\fi}}
\expandonce\a\expandonce\a
\long\gdef\b{afoono}

\endinput

-

Exhaustive (\edef-like) expansion is not possible with pdfTeX 1.40 in an expandable way. All suggested methods differ in their behavior from \edef:

• Using \csname fails whenever the exhaustive expansion contains non-character tokens (e.g. primitives or protected macros)
• Using \romannumeral expands only until the first unexpandable token, as explained by Joseph.
-
Good point about non-character tokens. As I said in the comment to Joseph, I'll bet \romannumeral can be used to do it. –  TH. Dec 21 '10 at 18:54
Actually, dealing with spaces and braces might be insurmountable. =/ –  TH. Dec 21 '10 at 19:25
@TH. I've added a solution which deals with simple cases, but as you say spaces will fail and braces get stripped out. As with many of these things, without engine support some things are not doable in an expandable manner. (TeX may be Turing complete, but that says nothing about expansion.) –  Joseph Wright Dec 21 '10 at 19:45
I've added a solution which deals with the 'stopping at first unexpandable token' case, but with lots of caveats. I'd agree with you that there is no general solution without \expanded. –  Joseph Wright Dec 21 '10 at 19:47
\catcode @=13
\def\at{@}

That expands to 13`. I think you misunderstood what I'm after. Take a look at the example in the question. –  TH. Dec 21 '10 at 19:23