# Expand multiple math-tokens before detokenize

I am a long time Latex user and over the last few weeks I am involved in a project that needs to do the following: I want to put in verbatim some math latex code that may include user-defined macros. Let's make an example:

\documentclass[a4paper,11pt]{article}
\usepackage{amsmath}
\def\userMacro{\sqrt{4}}
\begin{document}
\texttt{My Verbatim equation: \expandafter\detokenize\expandafter{\userMacro}}
\end{document}


Which renders:

My Verbatim equation:  \sqrt {4}


Now my issues comes when instead of a sigle macro, I have more than one, as in the following case:

\documentclass[a4paper,11pt]{article}
\usepackage{amsmath}

\def\userMacro{\sqrt{4}}

\begin{document}
\texttt{My Verbatim equation: \expandafter\detokenize\expandafter{\userMacro\userMacro}}
\end{document}


The output is:

My Verbatim equation:  \sqrt {4}\userMacro


Any idea how to expand an arbitrary number of token before applying

\detokenize


Thanks!

EDIT 06/01/2018: I should have reinforced the fact that the

math latex code that may include user-defined macros

So, the best example would be:

\documentclass[a4paper,11pt]{article}
\usepackage{amsmath}

\def\userMacro{\sqrt{4}}
\def\R{\mathbb{R}}
\begin{document}
\texttt{My Verbatim equation: \expandafter\detokenize\expandafter{\int_{\R}\userMacro}}
\end{document}


Should render as

My Verbatim equation:  \int_{\mathbb{R}}\sqrt{4}


Hope this clarifies. If you consider this to be a completely different question, I'll accept the best answer and eventually pose the question again.

Apologies.

EDIT 2:

\newcommand\xdetokenize[1]{\detokaux#1\edetok\edetok}
\def\detokaux#1#2\edetok{\ifx\edetok#1\edetok\else
\expandafter\IfSubStr*{\detokenize\expandafter{#1}}{protect}{\detokenize{#1}}{\detokenize\expandafter{#1}}\def\next{\detokaux#2\edetok\edetok}\expandafter\next\fi
}


With this tweak (using xstring) I am able to avoid the "protected macros" expansion. I think this is enough for the purpose of this question. Thanks to everyone!

• are the "tokens" guaranteed to be parameterless macros? what does "expanding an arbitrary number of tokens" actually mean? does it mean for example expanding once the first N tokens with a given N? and then does { count as a token? etc... – user4686 May 31 '18 at 21:37

In the general case (without knowing the exact composition of the full argument to \detokenize), you have to expand your way into the argument.

To dissect this example below, the 1st, 3rd, and 4th \expandafters expand the 2nd macro once, leaving the 2nd \expandafter to finally expand the 1st macro once.

Note that no \expandafter is needed before the \detokenize.

\documentclass[a4paper,11pt]{article}
\usepackage{amsmath}

\def\userMacro{\sqrt{4}}

\begin{document}
\texttt{My Verbatim equation:
\detokenize\expandafter\expandafter\expandafter{\expandafter\userMacro\userMacro}}
\end{document}


Now...

if you can guarantee that the argument will only contain macros that you want detokenized exactly once, then this will automate it:

\documentclass[a4paper,11pt]{article}
\usepackage{amsmath}
\let\edetok\relax
\def\userMacroA{\sqrt{4}}
\def\userMacroB{\frac{A}{B}}
\def\userMacroC{\qed}
\newcommand\xdetokenize[1]{\detokaux#1\edetok\edetok}
\def\detokaux#1#2\edetok{\ifx\edetok#1\edetok\else
\detokenize\expandafter{#1}\def\next{\detokaux#2\edetok\edetok}\expandafter\next\fi
}

\begin{document}
\ttfamily My Verbatim equation:

1: \xdetokenize{}

2: \xdetokenize{\userMacroA}

3: \xdetokenize{\userMacroA\userMacroB}

4: \xdetokenize{\userMacroA\userMacroB\userMacroC}

\end{document}


If you put plain text into the argument of \xdetokenize, the spaces will be eaten up (I warned you). It could be adapted to avoid that, but becomes increasingly complex to do so.

• Note that this is a particular behaviour of a few primitives, here \detokenize. – Manuel May 31 '18 at 20:16
• Issue: if I use xdetokenize on standard math commands, it doesn't work properly. Meaning, it expands "too much". Try, for example, \texttt{\xdetokenize{\frac{A}{B}}} – MBerra Jun 1 '18 at 8:05
• I understand that the above comment is not a good one. You warned about the "exactly one" expansion. And this is what your code does. I should have been more specific with my question. Try to edit it now. – MBerra Jun 1 '18 at 8:16
• @MBerra You are understanding the issue with expansion. Some things require multiple expansions, others are already fully expanded. Just for clarification, what exactly would you expect from \xdetokenize{\frac{A}{B}} anyhow? – Steven B. Segletes Jun 1 '18 at 15:31
• @StevenB.Segletes apologies for the late answer. I expect xdetokenize to leave the argument as it is. – MBerra Jun 5 '18 at 16:09

So you have a snippet of text (or whatever set of tokens) which is interspersed by some tokens that need to be expanded to whatsoever level.

A homespun and old fashioned approach to the matter could be splitting that snippet of text up into pieces whereof each piece begins with a token that is to be expanded.
Then have these pieces nested in reverse order.
Then with each piece have expanded the first token as many times as needed and then have that piece exchanged with the pieces that are nested behind it. This means reversing the reversed order after expanding. This means obtaining the initial order while things are expanded.

Before elaborating on that and delivering a minimal working example, let's in advance have some things related to (La)TeX explained.

First thing:

In (La)TeX there are so-called primitives. Some of them process so called ⟨general text⟩. E.g., \detokenize doses so. The Syntax of \detokenize is:

\detokenize<general text>


while

 <general text>=<filler><catcode-1-token><balanced text><explicit character token of catcode 2>


. Thus the syntax of \detokenize can also be described as:

\detokenize<filler><catcode-1-token><balanced text><explicit character token of catcode 2>


⟨filler⟩ is an arbitrary set of space tokens and tokens whose meaning equals the meaning of the \relax-primitive. ⟨filler⟩ will be removed.

⟨balanced text⟩ is an arbitrary set of tokens containing explicit catcode-1(begin group) character tokens and explicit catcode-2(end group) character tokens in amounts that are equal to each other.

\detokenize will extract ⟨balanced text⟩ from ⟨general text⟩ and act on that ⟨balanced text⟩ as if it were written to external file unexpanded and read back and tokenized from that external file under a catcode-régime where the space token has category code 10(space) and all other characters have category code 12(other).

The funny thing is:

When (La)TeX searches for the ⟨catcode-1-token⟩ preceding the ⟨balanced text⟩, it will both keep removing tokens forming the ⟨filler⟩ (i.e., space tokens and tokens whose meaning equals the meaning of the \relax primitive) and keep expanding exandable tokens until finding the desired ⟨catcode-1-token⟩.

The fact that (La)TeX keeps expanding while removing ⟨filler⟩ and searching for that desired ⟨catcode-1-token⟩ can be used for (e.g., via \expandafter and \romannumeral-expansion) triggering a lot of expansion work before letting (La)TeX find that desired ⟨catcode-1-token⟩.

Second thing:

You can use the expandable \romannumeral-primitive for triggering a lot of expansion work: \romannumeral usually causes (La)TeX to gather a ⟨number⟩ and deliver its representation in terms of lowercase roman numerals. While gathering tokens considered (digits of) ⟨numbers⟩, (La)TeX keeps expanding expandable tokens. In case the ⟨number⟩ gathered is not positive, the tokens forming that ⟨number⟩ will be "eaten" by (La)TeX while in that case (La)TeX will not deliver any token at all. Therefore one can (ab?)use the \romannumeral-primitive for triggering a lot of expansion-work (e.g., argument-exchanging-work, token-replacing-work) as long as it is ensured that in the end (La)TeX will find a non-positive ⟨number⟩.

Third thing:

Usually you use \expandafter for influencing the order in time in which expandable tokens get expanded.

E.g., if you wish the second token of the arrangement of tokens which is formed by the input-stream in chronological order to be expanded before the first token of the input-stream, you just make sure that within the input-stream the first token will be preceded by \expandafter:

\def\first{1}
\def\second{2}

\expandafter\first\second
-> \first 2


\expandafter itself is expandable. Referring to the analogy in the TeXbook, where TeX' processing of input and tokens is compared to a digestive tract where expansion takes place in TeX' mouth, one could say that \expandafter is processed in TeX' mouth. When work triggered by \expandafter is done, the \expandafter-token in question will not be there any more. (La)TeX considers the work triggered by \expandafter done when the process of attempting to expand the next but one token is finished. Therefore you can intersperse the first k elements of a set of tokens with \expandafter in order to have (La)TeX expand the (k+1)-th token before further processing the first token of that set. Such interspersing is called creating an \expandafter-chain:

\def\first{1}
\def\second{2}
\def\third{3}
\def\fourth{4}

\expandafter\first\expandafter\second\expandafter\third\fourth
-> \first\second\third 4.


You can have several \expandafter-chains.

For each additional level of expansion with a specific token as starting-point, you need to add another \expandafter-chain "leading" towards that token. The amount of \expandafter-tokens needed increases exponentially with the amount of expansion-levels desired:

 \def\first{1}
\def\second{2}
\def\third{3}
\def\fourth{4}
\def\leveluponfourth{\fourth}

%|1st \expand|2nd \expand|%
%|after-chain|after-chain|%
%|     |     |     |     |%
\expandafter\expandafter
\expandafter            \first
\expandafter\expandafter
\expandafter            \second
\expandafter\expandafter
\expandafter            \third
\leveluponfourth
->
\expandafter\first
\expandafter\second
\expandafter\third
\fourth
->
\first\second\third 4


You can avoid inserting many \expandafter-chains by using \romannumeral-expansion for implementing a mechanism whose syntax is
\romannumeral\Expandtimes{<number whose value is k>}<token-sequence>
and which delivers k "hits" by \expandafter to the first token of ⟨token-sequence⟩ as a result of triggering expanding the \romannumeral-primitive. Expanding the \romannumeral-primitive in turn can, e.g., be triggered by a single \expandafter-chain "leading" towards that \romannumeral-primitive.
In expansion contexts only one \expandafter-chain leading to the \romannumeral-primitive is needed for obtaining the same result that is obtained by k \expandafter-chains leading to the first token of the ⟨token-sequence⟩:

 \def\first{1}
\def\second{2}
\def\third{3}
\def\fourth{4}
\def\leveluponfourth{\fourth}

\expandafter\first
\expandafter\second
\expandafter\third
\romannumeral\Expandtimes{2}\leveluponfourth
->
\first\second\third 4


There are several ways of implementing \Expandtimes. One of them is shown in the minimal working example at the end of this answer.

After having explained these things (or at least after having tried to), let's go back to my homespun and old fashioned approach to the matter:

Assume you have the sequence

Whatsoever before tokenA.\tokenA Whatsoever between
tokenA and tokenB.\tokenB Whatsoever between tokenB
and tokenC.\tokenC Whatsoever between tokenC and
tokenD.\tokenD Whatsoever after tokenD.


, and you wish
\tokenA to be expanded 3 times ,
\tokenB to be expanded 1 times ,
\tokenC to be expanded 2 times ,
\tokenD to be expanded 4 times.

First split that snippet of text up into pieces whereof each piece begins with a token that is to be expanded:

{Whatsoever before tokenA.}%
{\tokenA Whatsoever between tokenA and tokenB.}%
{\tokenB Whatsoever between tokenB and tokenC.}%
{\tokenC Whatsoever between tokenC and tokenD.}%
{\tokenD Whatsoever after tokenD.}%


Then have these pieces nested in reverse order:

{\tokenD Whatsoever after tokenD.}{%
{\tokenC Whatsoever between tokenC and tokenD.}{%
{\tokenB Whatsoever between tokenB and tokenC.}{%
{\tokenA Whatsoever between tokenA and tokenB.}{%
Whatsoever before tokenA.%
}%
}%
}%
}%


Then with each piece have expanded the first token as many times as needed and also have the piece exchanged with the pieces that are nested behind it. In the minimal working example below, all this is done by means of the macro \Expandtimesexchange whose first argument denotes the level of expansion needed with the first token of its second argument before exchanging its second and its third argument:

\Expandtimesexchange{4}{\tokenD Whatsoever after tokenD.}{%
\Expandtimesexchange{2}{\tokenC Whatsoever between tokenC and tokenD.}{%
\Expandtimesexchange{1}{\tokenB Whatsoever between tokenB and tokenC.}{%
\Expandtimesexchange{3}{\tokenA Whatsoever between tokenA and tokenB.}{%
Whatsoever before tokenA.%
}%
}%
}%
}%


In case you wish all this to be done before applying \detokenize, you can – via inserting \expandafter right behind \detokenize – trigger some \romannumeral-expansion which will be terminated when arguments are expanded and exchanged around in a way where as a result of expansion \romannumeral will find the non-positive number 0 - the space token behind the zero "lets (La)TeX know" that the digit-sequence to be gathered is complete. Thus that space token will be "eaten" as part of completing the "gathering-the-digits-of-a-number"-process triggered by \romannumeral:

\detokenize\expandafter{%
\romannumeral\Expandtimesexchange{4}{\tokenD Whatsoever after tokenD.}{%
\Expandtimesexchange{2}{\tokenC Whatsoever between tokenC and tokenD.}{%
\Expandtimesexchange{1}{\tokenB  Whatsoever between tokenB and tokenC.}{%
\Expandtimesexchange{3}{\tokenA  Whatsoever between tokenA and tokenB.}{%
0 Whatsoever before tokenA.% "0<space>" ends \romannmeral expansion.
}%
}%
}%
}%
}%


Here is the promised complete minimal working example:

\documentclass[a4paper,11pt]{article}

\begingroup
\makeatletter
\@firstofone{%
\endgroup
\newcommand*\dfork[1]{\innerdfork#1{\@firstoftwo}d{\@secondoftwo}dd}%
}%
\newcommand\innerdfork{}\def\innerdfork#1d#2#3dd{#2}%
\newcommand*\Expandtimes[1]{%
0\expandafter\innerExp
\expandafter{%
\expandafter}%
\romannumeral\number\number#1 000d%
}%
\newcommand*\innerExp[2]{\dfork{#2}{#1 }{\innerExp{#1#1\expandafter}}}%

\newcommand*\Expandtimesexchange[2]{%
\expandafter\bracestripexchange\expandafter{\romannumeral\Expandtimes{#1}#2}%
}%
\newcommand\bracestripexchange[2]{#2#1}%

\newcommand\tokenA{\onelevelA}
\newcommand\tokenB{ [B.] }
\newcommand\tokenC{\onelevelC}
\newcommand\tokenD{\onelevelD}
\newcommand\onelevelA{\twolevelA}
\newcommand\onelevelC{ [C.] }
\newcommand\onelevelD{\twolevelD}
\newcommand\twolevelA{ [A.] }
\newcommand\twolevelD{\treelevelD}
\newcommand\treelevelD{ [D.] }

\begin{document}

\scriptsize
\parindent=0ex
\parskip=\baselineskip

\begin{verbatim}
\newcommand\tokenA{\onelevelA}
\newcommand\tokenB{ [B.] }
\newcommand\tokenC{\onelevelC}
\newcommand\tokenD{\onelevelD}
\newcommand\onelevelA{\twolevelA}
\newcommand\onelevelC{ [C.] }
\newcommand\onelevelD{\twolevelD}
\newcommand\twolevelA{ [A.] }
\newcommand\twolevelD{\treelevelD}
\newcommand\treelevelD{ [D.] }

\expandafter\newcommand
\expandafter\test
\expandafter{%<-these \expandafter trigger \detokenize
\detokenize\expandafter{%<- this \expandafter triggered by \detokenize triggers \romannumeral-expansion
\romannumeral\Expandtimesexchange{4}{\tokenD Whatsoever after \tokenD.}{%
\Expandtimesexchange{2}{\tokenC Whatsoever between \tokenC and \tokenD.}{%
\Expandtimesexchange{1}{\tokenB  Whatsoever between \tokenB and \tokenC.}{%
\Expandtimesexchange{3}{\tokenA  Whatsoever between \tokenA and \tokenB.}{%
0 Whatsoever before \tokenA.% "0<space>" ends \romannmeral expansion.
}%
}%
}%
}%
}%
}%
\end{verbatim}

\expandafter\newcommand
\expandafter\test
\expandafter{%<-these \expandafter trigger \detokenize
\detokenize\expandafter{%<- this \expandafter triggered by \detokenize triggers \romannumeral-expansion
\romannumeral\Expandtimesexchange{4}{\tokenD Whatsoever after \tokenD.}{%
\Expandtimesexchange{2}{\tokenC Whatsoever between \tokenC and \tokenD.}{%
\Expandtimesexchange{1}{\tokenB  Whatsoever between \tokenB and \tokenC.}{%
\Expandtimesexchange{3}{\tokenA  Whatsoever between \tokenA and \tokenB.}{%
0 Whatsoever before \tokenA.% "0<space>" ends \romannmeral expansion.
}%
}%
}%
}%
}%
}%

\texttt{\string\meaning\string\test $$\to$$ \meaning\test}

\bigskip

Now see that \texttt{\string\detokenize}-ing took place before definig:

\bigskip

\texttt{\string\test $$\to$$ \test}

\end{document}


• Wow, this is a really good elaboration on the \romannumeral-thing I've seen suggested multiple times but didn't use because I simply didn't understand what it was doing and why. After merely skimming this answer I still don't, but I will carefully study it later. – Peter Nerlich Aug 1 '18 at 14:28
• – Ulrich Diez Aug 1 '18 at 21:08

No \expandafter, but you have to be careful with the code you use inside \stringify.

Macros defined with \defineusermacro are safe; commands such as \frac aren't.

\documentclass[a4paper,11pt]{article}
\usepackage{amsmath}
\usepackage{xparse}

\ExplSyntaxOn
\NewDocumentCommand{\stringify}{m}
{
\group_begin:
\bool_set_true:N \l_stringify_expand_bool
\tl_to_str:x { #1 }
\group_end:
}
\cs_generate_variant:Nn \tl_to_str:n { x }
\bool_new:N \l_stringify_expand_bool

\NewDocumentCommand{\defineusermacro}{mm}
{
\newcommand{#1}
{
\bool_if:NTF \l_stringify_expand_bool
{ \exp_not:n { #2 } }
{ #2 }
}
}
\ExplSyntaxOff

\defineusermacro\userMacro{\sqrt{4}}

\begin{document}

\texttt{My Verbatim equation: \stringify{\userMacro+\userMacro}}

\end{document}


You can use xinttools facilities for things like this.

\documentclass[a4paper,11pt]{article}
\usepackage{amsmath}
\newcommand\userMacro{\sqrt{4}}
\newcommand\otherMacro{\frac{3}{7}}
\newcommand\ContainerMacro {%
\userMacro
\otherMacro
\userMacro
\otherMacro
\userMacro
\otherMacro
\userMacro
\otherMacro
\userMacro
\otherMacro
\userMacro
\otherMacro
}%

\usepackage{xinttools}

\begin{document}
\texttt{My Verbatim equation:
\xintApplyUnbracedNoExpand{\detokenize\expandafter}
{%
\userMacro
\otherMacro
\userMacro
\otherMacro
\userMacro
\otherMacro
\userMacro
\otherMacro
\userMacro
\otherMacro
\userMacro
\otherMacro
}%
}

\texttt{My Verbatim equation:
\xintApplyUnbraced{\detokenize\expandafter}
{\expandafter\space\ContainerMacro}%
}%

\end{document}


The \expandafter\space\ContainerMacro expands once \ContainerMacro, then the \space will stop the so-called "f"-expansion done by \xintApplyUnbraced to its argument.

Another macro of xinttools called \xintApplyInline fetches too token by token but it "f"-expands each item, so it can't be used here.