I was surprised the use of \detokenize
was not shown, but I then noted that egreg had it at one point until Phillipe pointed out that
And a drawback you don't mention of \detokenize is that it inserts spaces after macros names with more than one character (not sure if it's relevant for regular expressions).
The tokcycle
package, as part of its examples document provides an \altdetokenize
which, despite its name, is not an alternate way to detokenize. Rather, it takes detokenized output and reformulates its appearance to remove those ugly spaces that are inserted after macros, when it is appropriate to do so (through strategically placed \unskip
s).
In the MWE, first \detokenize
, then \altdetokenize
as described in the tokcycle
examples document: https://ctan.org/tex-archive/macros/generic/tokcycle. Note that T1
font encoding must be active for backslashes to render properly.
This is an interesting macro in that, unlike the recommended tokcycle
approach of collecting the processed tokens from the input in a token list and then outputting the token list at the end, this macro works in real-time, outputting its result during, rather than after, the token cycle. This is possible because the output is detokenized, and I don't have to worry about macros in the input stream demanding arguments that have not yet been processed.
\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{tokcycle}
\newif\ifmacro
\newcommand\altdetokenize[1]{\begingroup\stripgroupingtrue\macrofalse
\stripimplicitgroupingcase{-1}%
\tokcycle
{\ifmacro\def\tmp{##1}\ifcat\tmp A\else\unskip\allowbreak\fi\macrofalse\fi
\detokenize{##1}\ifx##1\bgroup\unskip\fi\ifx##1\egroup\unskip\fi}
{\ifmacro\unskip\macrofalse\fi\{\processtoks{##1}\ifmacro\unskip\fi\}\allowbreak}
{\tctestifx{\\##1}{\\}{\ifmacro\unskip\allowbreak\fi
\allowbreak\detokenize{##1}\macrotrue}}
{ \hspace{0pt plus 3em minus .3ex}}
{#1}%
\unskip
\endgroup}
\begin{document}
\detokenize{[RegularExpression(@"\d+")]}
\altdetokenize{[RegularExpression(@"\d+")]}
\end{document}
