Is it possible to fully expand tokens in an expandable manner and preserves category codes? I'd like to do this just using pdfTeX.

I'm looking for something that would work like this:

\fullyexpand{\foo bar baz}

which would expand to the 10 tokens "foobar baz" and retain the category codes.

If we drop the expandable requirement, then this is easy.


If we drop the retain catcode requirement, it's doable, but less straight-forward. Here's the best I could come up with.


I was hoping to combine the ε-TeX extension \scantokens with the above (by an appropriate, trivial modification of \gobbleprefix), but of course, that does not work. \scantokens's expansion is empty and pdfTeX just acts like it has opened a new file.

(As an aside, it amuses me that not expanding tokens is expandable—using \unexpanded—but expanding tokens isn't, at least not obviously so.)

  • I will write an answer to this question today or tomorrow. In the meantime, you might be interested in \expandsome{\foo{ } \bar{\expandthis\foo}\expandthis\bar{abc}} which will expand precisely the tokens following \expandthis, in an expandable manner. The code can be found online, or at this question. --- Actually, it would help me if someone has a list of all primitives that should be expanded. Feb 15, 2011 at 9:32

4 Answers 4


Did you try using \romannumeral? This is used a lot for this type of thing (see for example the \exp_args:Nf concept in expl3):

\def\fullyexpand#1{\romannumeral - `0#1}

This works because TeX will keep expanding #1 looking for a number, which will always turn out to be negative, so the Roman numeral will vanish. Note that this solution will stop on the first non-expandable token, unlike an \edef which will keep going.

It's possible to build a function which can expand using \romannumeral 'around' unexpandable tokens. For example, the following code will work reasonably well:

  \csname donothing\fullyexpandauxi{#1}{}%
  \expandafter\fullyexpandauxii\romannumeral -`0#1\fullyexpandend

However, this is not the same as \expanded, for a few reasons. First, my implementation will strip out spaces in the argument (as it does a loop, and TeX will skip spaces). Braces will also get stripped out. A bit of testing also reveals that \romannumeral will expand \protected functions here, whereas \expanded does not. I'd also note that the above code needs some guards adding for a blank (empty or all space) argument, as currently things fail in these cases.

With current release LuaTeX one can use \expanded, which does more-or-less the same as an \edef but is expandable (it doesn't required doubled # tokens also). This primitive will be in TeX Live 2019 pdfTeX/e-pTeX/e-upTeX, and hopefully in XeTeX (yet to be confirmed). As a precursor to this, expl3 has a macro-based emulation, slow but working, which does token-by-token examination and allows 'e-type' expansion.

On the aside, it is possible to use \scantokens expandably, but as you may have found this can be tricky and it is usually necessary to have a (non-expandable) change of \everyeof first. LuaTeX addresses this issue with the \scantextokens primitive, which combines this end-of-file stuff directly into the primitive. Of course, if you are using LuaTeX then the original problem is solvable anyway, since \expanded is available.

  • Wouldn't it also strip out leading numbers?
    – TH.
    Dec 21, 2010 at 18:11
  • @TH.: no, because -`0 is already a complete number, only that is stripped.
    – Philipp
    Dec 21, 2010 at 18:16
  • @Joseph Wright: Exhaustive expansion in LuaTeX doesn't need \directlua because even current versions of LuaTeX have \expanded.
    – Philipp
    Dec 21, 2010 at 18:25
  • @Philipp: Oh, I see. I didn't even notice the backtick. It's interesting that TeX allows an optional space after numbers of that form. I guess it is expanding looking for the space, but that is very strange. Yeah, I guess it is: \romannumeral 5\space 3 produces v3. I'll bet this can be leveraged to expand everything.
    – TH.
    Dec 21, 2010 at 18:52
  • @Philipp @Joseph For Roman numerals TeX defines an empty expansion, if the number is zero or negative why do you need the minus?
    – yannisl
    Dec 21, 2010 at 18:56

After learning many tricks on this forum, I propose the following solution. I think it does what @TH. wanted, namely it expands everything, and when it meets a non-expandable token, it just stores it and continues. It uses a future package of mine, whose code can currently be found online, or as my answer to this question. This code is input in the first line below. Set whichever filename you want.

Note that I have not been careful to reset catcodes to their default value in ULcase.sty, so here, we don't need \catcode`:=11\catcode`_=11\relax. One day, I'll clean this up.

% Code based on the extended Upper- and Lower-casing code found
% in the ULcase package.
\input ULcase.sty\relax

% ============ Table |fullyexpand|, |\fullyexpand|
% The |fullexpand| table expands every token that is expandable
% according to the test \FE_token_if_expandable:NTF.
% Then |\fullyexpand| is basically changing the case using a special
% "case table", |ULfullyexpand|. Here, |\MEA_trigger:f| is |\romannumeral|
% in disguise, forcing the full expansion of |\UL_to_case_aux:nn|
% (This is necessary for technical reasons.)

% A few tests, building up to \FE_token_if_expandable:NTF.
  \ifdefined #1%
% |\expandsome| only expands the tokens following |\expandthis|.
% |\expandafter:nw{...}\foo| expands |\foo| before |...|.

% ===== Building the table.
% We copy the standard definitions (in particular for braces)
% NB: maybe problem with \NoCaseChange.
% Spaces are just kept:
\UL_setup:nnn{ULfullyexpand}{ }{ }
% The default action is to check if #3 (the next token) is expandable.
% If it is, we expand it. Otherwise, we output it. "#2{#3}" is responsible
% for continuing the loop.
% Define |\noexpand|
\long\expandafter\gdef\csname UL_table_ULfullyexpand_%
% Define |\detokenize|
\long\expandafter\gdef\csname UL_table_ULfullyexpand_%
% Define |\unexpanded|
\long\expandafter\gdef\csname UL_table_ULfullyexpand_%
% A helper for |\detokenize| and |\unexpanded|.

% ===== Tests
\long\gdef\fooA#1{\fooC{\noexpand\fooA got #1} \fooB{a nice #1}}
\long\gdef\fooB#1{fooB\space got <#1>}
\protected\def\fooC#1{Not expanded.}
\long\gdef\a{\fullyexpand{Text: \fooA{argument}}}
\long\gdef\b{Text: \fooC {\fooA got argument} fooB got <a nice argument>}

\long\xdef\b{Text:\detokenize{\fooC {\noexpand \fooA got argument} %
    \fooB {a nice argument}\fooA {hi}}}

\long\xdef\b{Text:\unexpanded{\fooC {\noexpand \fooA got argument} %
    \fooB {a nice argument}\fooA {hi}}}

\long\gdef\a{\fullyexpand{a\foo \ifnum1=1\number2 yes\else no\fi}}


Exhaustive (\edef-like) expansion is not possible with pdfTeX 1.40 in an expandable way. All suggested methods differ in their behavior from \edef:

  • Using \csname fails whenever the exhaustive expansion contains non-character tokens (e.g. primitives or protected macros)
  • Using \romannumeral expands only until the first unexpandable token, as explained by Joseph.
  • Good point about non-character tokens. As I said in the comment to Joseph, I'll bet \romannumeral can be used to do it.
    – TH.
    Dec 21, 2010 at 18:54
  • Actually, dealing with spaces and braces might be insurmountable. =/
    – TH.
    Dec 21, 2010 at 19:25
  • @TH. I've added a solution which deals with simple cases, but as you say spaces will fail and braces get stripped out. As with many of these things, without engine support some things are not doable in an expandable manner. (TeX may be Turing complete, but that says nothing about expansion.)
    – Joseph Wright
    Dec 21, 2010 at 19:45
  • 1
    I've added a solution which deals with the 'stopping at first unexpandable token' case, but with lots of caveats. I'd agree with you that there is no general solution without \expanded.
    – Joseph Wright
    Dec 21, 2010 at 19:47

How about this?

\catcode `@=13
  • That expands to 13. I think you misunderstood what I'm after. Take a look at the example in the question.
    – TH.
    Dec 21, 2010 at 19:23
  • @TH Quite possibly! Will have another look:)
    – yannisl
    Dec 21, 2010 at 19:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .