# Detokenizing without extra spaces?

I have a need to pass through potentially arbitrary characters untouched and found this macro:

\makeatletter
\def\test#1{\expandafter\zap@space\detokenize{#1} \@empty}
\makeatother


The problem is that \detokenize inserts spaces after commands in the expansion so I get rid of these with LaTeX's \zap@space. Unfortunately, I need to keep any spaces which \detokenize did not produce. I suspect that there is some cunning way to do this by redefining the catcode of the space character but it is somewhat beyond me.

For example (yes, it's regular expressions I need to pass through ...),

\test{\A\d{2,}.+\z}


Should expand to \A\d{2,}.+\z

but

\test{A test}


should expand to A test and not lose the space that was there originally.

I should say, that \verb doesn't work as these regexp strings are the value of a keyval pair:

\macro[key=\test{#1}]


Update It seems too hard to do this in general without making other things rather messy so I've settled for requiring that any regexps used are canonicalised to use no literal spaces in them. It's always possible to replace them with '\x20' for example in anyway so this means that \zap@space will only be zapping spaces created by \detokenize, which is fine. Many thanks for all of the answers though as they are very instructive.

• You should add more details about what you need; if it is only about printing, then \verb|\A\d{2,}.+\z| would be sufficient. – egreg Feb 13 '12 at 13:42
• You're right, I have added a bit more context. – PLK Feb 13 '12 at 13:51
• I don't think there is an easy way to do this. You would need to scan for the backslash followed by characters and only then remove the space -- possible but not trivial. Maybe if you code your \test macro in a way that allows to declare the regexes before using them? \declareregex\myregex{\A\d{2,}.+\z} (which reads the argument verbatim) and then \macro[key=\test{\myregex}] – Martin Scharrer Feb 13 '12 at 14:07
• I can search out an old solution I made sometime back, but before then let us know if you need to retokenize the outcome of detokenizing. – Ahmed Musa Feb 13 '12 at 15:46
• No, I don't need to retokenize. The raw characters are going elsewhere, not TeX. – PLK Feb 13 '12 at 17:21

It is impossible to distinguish \A . from \A. once TeX has converted those into tokens: the only solution if you need to preserve those spaces is to read the argument verbatim.

However, if you are fine with that, then the simplest method is to update the l3kernel and l3experimental bundles (and l3packages) to a very recent version (Februrary 2012), then use tools from the l3regex package to add \string in front of each token in the argument, and expand. The code below does that (replace \tl_show:N by whatever you want to do to the string).

\documentclass{article}
\usepackage{l3regex}
\ExplSyntaxOn
\cs_new_protected:Npn \test #1
{
\tl_set:Nn \l_tmpa_tl {#1}
\regex_replace_all:nnN { . } { \c{string} \0 } \l_tmpa_tl
\tl_set:Nx \l_tmpb_tl { \l_tmpa_tl }
% now \l_tmpb_tl contains what you want:
\tl_show:N \l_tmpb_tl
}
\ExplSyntaxOff
\begin{document}
\test{\A\d{2,}.+ Hello, world!\z}
\end{document}


How does it work? \regex_replace_all:nnN performs a replacement on a stored token list, so we need to store the argument.

\tl_set:Nn    % Set locally
\l_tmpa_tl  % the "local temporary token list" \l_tmpa_tl
{#1}        % to contain "#1" (the argument).
\regex_replace_all:nnN % Replace every occurrence of
{ . }                % any token, even braces etc.
{                    % by
\c{string}         %   \string
\0                 %   what was matched (the token)
} \l_tmpa_tl         % in \l_tmpa_tl
\tl_set:Nx        % Set locally, with expansion,
\l_tmpb_tl      % the "local temporary token list b"
{ \l_tmpa_tl }  % to (the expansion of) \l_tmpa_tl
\tl_show:N    % Show the contents of
\l_tmpb_tl  % the token list variable \l_tmpb_tl


Of course, under the hood, l3regex does a lot of work so it will depend on how many such regular expressions you have to go through.

EDIT: A plain TeX solution for the very specific task your are asking for. I am assuming that the strings never contain the character ^^A (char code 1). The idea is to use \lowercase to change all true space tokens to some recognizable character. Then \detokenize, and loop through the result one character at a time (this automatically skips spaces) replacing ^^A by a space.

\catcode64=11
\long\def\test#1%
{%
\begingroup
% Ensure that every character is preserved by \lowercase.
\count@\z@
\loop\ifnum\count@<256
\lccode\count@\z@
\repeat
% Except spaces, changed to ^^A
\lccode32=\@ne
\lowercase
{%
\endgroup
\edef\result{\expandafter\test@\detokenize{#1}\relax}%
}%
}
% Then map {^^A => space, space =>} onto the string.
\def\test@#1%
{%
\ifx#1\relax\test@end\fi
\ifnum#1=\@ne\space\else#1\fi
\test@
}
\def\test@end\fi#1\test@{\fi}
\catcode64=12
\test{ab c\d e{f} \fg }\show\result

• Bruno: I get c{string}\c{string}Ac{string} c{string} ... Please what does it mean? – Ahmed Musa Feb 13 '12 at 23:23
• @AhmedMusa Update your system. – egreg Feb 13 '12 at 23:49
• @Bruno I used MiKTeX Package Manager less than 2 weeks ago. Which packages do I need to update? – Ahmed Musa Feb 14 '12 at 0:27
• @AhmedMusa What version is your l3regex.sty? Mine is l3regex.dtx 3326 2012-02-07 – egreg Feb 14 '12 at 0:46
• @AhmedMusa Since your version is too old, \c was not implemented yet, and is simpley intepreted as a c, so instead of \string, you get c{string}. This feature is very recent, may less than two weeks. I can check more carefully if you need. – Bruno Le Floch Feb 14 '12 at 2:24

Despite the question being tagged as tex-core I would like to point at xparse. It has the argument specification v that does the detokinization without spaces as far as I understand it. Form the manual:

Arguments of type “v” are read in verbatim mode, which will result in the grabbed argument consisting of tokens of category code 12 (“other”), except spaces, which are given category code 10 (“space”).

\DeclareDocumentCommand\foo{v}{\ttfamily #1}


And using it with

\foo!\A\d{2,}.+\z!


produces the same output as

\verb!\A\d{2,}.+\z!


There are no extra spaces introduced by xparse. In this sense the contents of argument #1 is “untouched”.

• The point is that the OP wants to use this as an argument to another command, which is not possible for this kind of commands. – egreg Feb 21 '12 at 17:03
• Thanks - I didn't really know about xparse. The big problem is that I really need this as a value of a keyval pair and this rules out anything verbatim-like. It's a really quite hard problem. I think I'll have to settle for not allowing spaces in the input and using \detokenize – PLK Feb 21 '12 at 21:27

A truly general solution seems difficult but here is my attempt.

Call \spaceparse with the argument to be detokenized and parsed for spaces after commands. The result of the parsing is available in the macro \result. You would need to call \result to see the outcome.

Because \detokenize doubles the hash character, we first reverse that action. If you don't require this default action, then use the star (*) form of \spaceparse.

You can copy this into a package and call the package.

\documentclass{article}
\usepackage{catoptions}
% No conflict with etoolbox.sty:
% \usepackage{etoolbox}
\makeatletter
\robust@def*\spaceparse{\cpt@testst\sp@ceparse}
\robust@def*\sp@ceparse#1{%
\begingroup
\edef\@tempa{\detokenize{#1}}%
\ifboolTF{cpt@st}{}{\s@expandarg\cpt@pophash\@tempa\@tempa}%
\edef\@tempa##1{##1\expandcsonce\@tempa\@space\cpt@nil}%
\edef\@tempb##1{\def##1####1\@space####2\cpt@nil}%
\@tempb\@tempb{%
\ifblankTF{##2}{%
\toks@\expandafter{\the\toks@##1}%
}{%
\countbackslash{##1}%
\ifnum\nr=\@ne
\xifinsetTF{\@car##2\relax\@nil}\cpt@oth@rchars{%
\toks@\expandafter{\the\toks@##1}%
}{%
\cptexpanded{\toks@{\the\toks@\unexpanded{##1}\@space}}%
}%
\else
\cptexpanded{\toks@{\the\toks@\unexpanded{##1}\@space}}%
\fi
\@tempb##2\cpt@nil
}%
}%
\@tempa{\toks@{}\@tempb}%
\edef\result{\the\toks@}%
\postgroupdef\result\endgroup
}
\robust@def*\countbackslash#1{%
\begingroup
\@tempcnta\z@
\def\@tempa##1{%
\def\@tempa####1##1####2\@nil{%
\ifblankTF{####2}{}{%
\@tempa####2\@nil
}%
}%
\@tempa#1##1\@nil
}%
\s@expandarg\@tempa\@backslashchar
\cptexpanded{\endgroup\def\noexpand\nr{\the\@tempcnta}}%
}
\makeatother


Tests:

\def\x{##1\A\d{2,}.+\z A B\\x y}
\expandafter\spaceparse\expandafter{\x}
\show\result

\spaceparse{xx\x{f} \fg x}
\show\result

\spaceparse{ab c\d e{f} \fg x}
\show\result

\spaceparse{#1\A\d{2,}.+\z A B}
\show\result

\begin{document}

\end{document}

• I will try this and report back, thanks. I may have to try to see how to convert it to use etoolbox instead of ltxtools though. – PLK Feb 14 '12 at 17:50
• ltxtools-base is available on CTAN but, if you want, I can remove the dependence on ltxtools-base. However, the solution will have to be longer. – Ahmed Musa Feb 14 '12 at 17:58
• I really appreciate the effort you have put into this already ... unfortunately I can't really use ltxtools as it conflicts with definitions in etoolbox which is a non-negotiable dependency ... – PLK Feb 14 '12 at 18:56
• @PLK I have replaced ltxtools-base with catoptions and confirmed that there is no command name clash with etoolbox. BTW, thanks for highlighting a command name clash between ltxtools-base and etoolbox. – Ahmed Musa Feb 14 '12 at 19:05
• Does this work with Lualatex? It seems to require pdflatex? – PLK Feb 15 '12 at 7:29

Expandable answer: \test below is defined in such a way that it can be used in any context: \edef\foo{\test{...}} works, and similarly \typeout{\test{...}}. This relies on undocumented experimental internal features of LaTeX3, and I only guarantee that it works with today's version of expl3 (it should work with versions from January 8th 2012 until...?).

\RequirePackage{expl3}
\ExplSyntaxOn
\cs_new_nopar:Npn \test
{ \tl_act:NNNnn \test_normal:nN \test_group:nn \test_space:n { } }
\cs_new:Npn \test_space:n #1 { \tl_act_output:n {~} }
\cs_new:Npn \test_normal:nN #1 #2
{ \exp_args:No \tl_act_output:n { \token_to_str:N #2 } }
\cs_new:Npx \test_group:nn #1 #2
{
\exp_not:N \exp_args:Nf \exp_not:N \tl_act_output:n
{ \exp_not:N \test { \iow_char:N \{ #2 \iow_char:N \} } }
}
\ExplSyntaxOff
\typeout{\test{\a b c}}
\typeout{\test{\d*?z{2,} ??\b\p{Lu}}}
\stop


All the magic recursion through brace groups, and finding spaces, is done by \tl_act:NNNnn. Don't ask.

• This seems to loose the space after \a, giving \ab in the log. However, it doesn't if you replace the \a with \!` (or any other non-letter case). – Joseph Wright Feb 16 '12 at 9:28
• @Joseph: see comments on my non-expandable answer and Ahmed's comments. Once TeX has tokenized, this space is lost for good; same thing for multiple spaces in a row. The only correct solution is to use verbatim. – Bruno Le Floch Feb 16 '12 at 10:08
• Many thanks for working so hard on this. I think what I may need to do is to try to change the LaTeX interface so that I can use verbatim ... I will see – PLK Feb 17 '12 at 22:09