# Expandable 'character scanning' command that preserves spaces

David Carlisle recently offered a clean and rather elegant TeX solution to a question about expandable 'character scanning' commands. Unfortunately, David's solution does not preserve spaces in the input string. Thus, given David's code:

\def\expandloop#1{\xloop#1\relax}
\def\xloop#1{%
\ifx\relax#1\else[#1]\expandafter\xloop\fi}


we see the same output whether we supply

\edef\zzzz{\expandloop{TEST1}}


or

\edef\zzzz{\expandloop{  T ES T    1 }}


Why does this happen?

Is there a solution that treats space characters the same way as any others in the input string?

-

The following shows the output of three loops.

1. As noted the original drops all space tokens, that is, tokens of catcode 10.
2. If the space character has catcode 12 (other) then it is treated as a normal character and not dropped.
3. The variant loop scans ahead looking for real spaces and making them safe, but as in @Ryan's answer TeX has already compressed adjacent space characters into single tokens so you only get one [ ] for each run of spaces. This version works with normal catcode 10 characters however.
4. Fourthly a variant of the first which inserts a \string expansion into the loop so that braces rather than brace groups are treated as single tokens. (As noted by Bruno the original three versions do not do that).

Which is most acceptable depends on what you really want this for.

\$ pdftex xloop2
This is pdfTeX, Version 3.1415926-2.3-1.40.12 (TeX Live 2011/Cygwin)
restricted \write18 enabled.
entering extended mode
(./xloop2.tex
2:[ ][ ][T][E][S][T][ ][ ][1]
3:[ ][T][E][S][T][ ][1]
3:[ ][T][ES][T][ ][1]
4:[T][{][E][S][}][T][1]
)
No pages of output.
Transcript written on xloop2.log.


\def\expandloop#1{\xloop#1\relax}

\def\xloop#1{%
\ifx\relax#1\else[#1]\expandafter\xloop\fi}

%%%%%%%%%%%%%%%%%%%%%%%%%%%

\edef\tmp{\def\noexpand\expandloopb##1{\noexpand\xloopb##1\relax\space \valign}}
\tmp

\def\xloopb#1 #2\valign{%
\ifx\relax#2\relax
\xxloopb#1%
\else
\xloopb#1\xloopb#2\valign
\fi}

\def\xxloopb#1{%
\ifx\relax#1\else[\ifx\xloopb#1\space\else#1\fi]\expandafter\xxloopb\fi}

%%%%%%%%%%%%%%%%%%%%%%%%%%%

\def\expandloopc#1{\xloopc#1!}

\def\xloopc{\expandafter\xxloopc\string}

\def\xxloopc#1{%
\ifx!#1\else[#1]\expandafter\xloopc\fi}

%%%%%%%%%%%%%%%%%%%%%

{\catcode\ =12
\immediate\write20{2:\expandloop{  TEST  1}}}

\immediate\write20{3:\expandloopb{  TEST  1}}

\immediate\write20{3:\expandloopb{  T{ES}T  1}}

\immediate\write20{4:\expandloopc{  T{ES}T  1}}

\bye

-
You should really note that none of those answers will go through braces, contrarily to Joseph's and Ryan's answers. –  Bruno Le Floch Jul 8 '12 at 13:31
@BrunoLeFloch, true. noted that in the answer and added a version that \string processes the tokens –  David Carlisle Jul 8 '12 at 14:07

Possibly overkill, but within the LaTeX3 code base Bruno Le Floch has developed the idea of a 'token list action'. This is a generic concept which can then be applied in different scenarios, for example the problem posed in Are there purely expandable variants of \MakeUppercase?.

We have not provided a public interface for the generic mechanism, as it's not clear that this is needed. So here I've recoded the approach using standard TeX coding. As LaTeX3 requires \pdfstrcmp or equivalent functionality, I'm loading the pdftexmcds package to provide \pdf@strcmp.

\RequirePackage{pdftexcmds}
\makeatletter
\long\def\tl@action#1#2#3#4#5{%
\ifnum\iffalse{\fi}=\z@\fi
\tl@action@loop#5\q@action@mark\q@action@stop
{#4}#1#2#3%
\tl@action@result{}%
}
\long\def\tl@action@loop#1\q@action@stop{%
{\tl@action@normal}
{%
{\tl@action@group}
{\tl@action@space}%
}%
#1\q@action@stop
}
\long\def\tl@action@normal#1#2\q@action@stop#3#4{%
\ifx\q@action@mark#1%
\expandafter\tl@action@end
\fi
#4{#3}#1%
\tl@action@loop#2\q@action@stop
{#3}#4%
}
\long\def\tl@action@end#1\tl@action@result#2{%
\ifnum{=\z@}\fi
\z@
#2%
}
\long\def\tl@action@group#1#2\q@action@stop#3#4#5{%
#5{#3}{#1}%
\tl@action@loop#2\q@action@stop
{#3}#4#5%
}
\expandafter\long\expandafter\def\expandafter
\tl@action@space\space#1\q@action@stop#2#3#4#5{%
#5{#2}%
\tl@action@loop#1\q@action@stop
{#2}#3#4#5%
}
\long\def\tl@action@output#1#2\tl@action@result#3{%
#2%
\tl@action@result{#3#1}%
}
\def\q@action@mark{\q@action@mark}
\ifnum\pdf@strcmp
{\unexpanded\expandafter{\@firstofone#1{}}}{\unexpanded{#1{}}}=\z@
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
}
\ifcat*\expandafter\@gobble\expandafter{\expandafter{\string#1?}}*%
\expandafter\@secondoftwo
\else
\expandafter\@firstoftwo
\fi
}
\long\def\tl@wrap#1{%
\unexpanded\expandafter{%
\romannumeral
\tl@action
\tl@wrap@normal
\tl@wrap@normal
\tl@wrap@space
{}
{#1}%
}%
}
\long\def\tl@wrap@normal#1#2{%
\tl@action@output{[#2]}%
}
\long\def\tl@wrap@space#1{%
\tl@action@output{[ ]}%
}
\makeatother
\documentclass{article}
\begin{document}
\makeatletter
\tl@wrap{abc de  { } f}
\makeatother
\end{document}


In many ways, the idea of carefully scanning for spaces and groups is the same as Ryan's, although clearly the implementation here is a little different. (\tl@action can be used for many things, hence the empty and repeated arguments!)

On the 'why does this happen?' part of the question, there are two things to bear in mind. First, TeX will read multiple spaces as a single space. So something like

foo  bar  baz


will be read as

foo bar baz


unless something like \obeyspaces is in action. The latter has to be set up before reading material, and is not expandable.

Secondly, a macro of the form

\def\foo#1{<do stuff>}


will skip spaces after \foo when reading #1. Thus

\foo tokens


will skip the space after \foo, and read t as the first token. To avoid this, careful control of tokens is requires so that this does not happen.

-
Of course, if your input looks like foo bar with two spaces, TeX will convert to a single space before the code above gets to work. If you want to avoid that, you'll need to use a non-expandable route. –  Joseph Wright Jul 8 '12 at 9:50
What is wanted for material inside a brace group is not clear. I've assumed that {foo} should end up as [foo], but it is easy to alter this to give [[f][o][o]] or similar if required. –  Joseph Wright Jul 8 '12 at 13:49

In TeX, when a macro is being expanded and its arguments gathered, all intervening spaces are ignored. Thus, \xloop just hops from one letter to the next and eats all the space. I wrote an answer once that presented a fully expandable macro for "sanitizing" text: completely removing control sequences and braces. You can modify that answer to, say, put [] around each token by making the following change: replace \SanitizeTokens, as given there, with the following macro:

\newcommand\SanitizeTokens[1]{%
\ifx\SanitizeStop#1%
\else
\ifx\SanitizedSpace#1%
\RealSpace
\else
\ifx\ #1%
\RealSpace
\else
\if\relax\noexpand#1%
[\string#1]%
\else
[#1]%
\fi
\fi
\fi
\expandafter\SanitizeTokens
\fi
}


This will have the following effect:

\Sanitize{TEST1} = [T][E][S][T][1]
\Sanitize{ T E S T   1} =  [T] [E] [S] [T] [1]
\Sanitize{T{EST}1} = [T][E][S][T][1]
\Sanitize{TEST\macro} = [T][E][S][T][\macro]


Note that multiple spaces are condensed. This is because TeX itself combines repeated spaces into a single space when parsing input. To get around that you'd have to play tricks with catcodes and that's not expandable. It would be possible to get the curly braces to be bracketed as well, if that's an issue for you. I don't know if it's advisable. Finally, I've opted to force macro names not to expand; you can change the line with \string` in it if you prefer otherwise.

N.B. You may be wondering whether the outrageously complex macro I wrote is actually necessary for this simple change. The answer is yes: all the complexity is carefully tiptoeing around the macro parser to avoid losing spaces and being fooled by groups.

-