Parse a string into tokens of numbers and not numbers

I have a string that I want to parse into to numbers and non-numbers.

For my purposes:

A Number can EITHER be any sequential string of digits OR sequential string of digits with a . followed by another sequential string.

A Non-Number is anything that is not a Number.

For example

ljksadflh23898129hfafh0324.22234


should be parsed into something like

ljksadflh, 23898129, hfafh, 0324.22234


or

ljksadflh/23898129/hfafh/0324.22234


or whatever floats your boat as long as the list retains the same ordering.

-
I believe one could just insert a deliminator at every first instance of a number then split the string(if a . occurs before a number it would not be inserted). I would then need to convert the stringed numbers into usable numbers – Uiy Mar 16 '12 at 13:11
Do you care about edge conditions like 123.456.789? It seems it could parse to either 123.456 / . / 789 or 123 / . / 456.789 – StrongBad Mar 16 '12 at 13:42
no, not really. – Uiy Mar 16 '12 at 18:57

With the experimental (but pretty much ready for release) package l3regex (found in the l3experimental bundle on CTAN), this task is a piece of cake.

\documentclass{article}
\usepackage{l3regex,xparse}
\ExplSyntaxOn
\seq_new:N \l_uiy_result_seq
\NewDocumentCommand { \UiySplit } { m }
{
\regex_extract_all:nnN { \D+ | \d+(?:\.\d*)? } {#1} \l_uiy_result_seq
\seq_map_inline:Nn \l_uiy_result_seq { item:~##1\par }
}
\ExplSyntaxOff
\begin{document}
\end{document}


The \regex line splits the user input #1 into pieces which either consist of one or more (+) non-digits (\D), or (|) of one or more digits (\d), followed maybe (? acting on the group (...), which we want to be "non-capturing", done using (?:...)) by a dot (\. escaped dot, because the dot has a special meaning) and zero or more digits (\d*). The line below maps through all the matches we found, with ##1 being a single match. Of course, you can do whatever you want to do with the items of the sequence \l_uiy_result_seq.

Edit: The module also provides regular expression replacements. If I remember the syntax correctly, the following should work.

\ExplSyntaxOn
\seq_new:N \l_uiy_result_seq
\NewDocumentCommand { \UiySplit } { m }
{
\tl_set:Nn \l_uiy_result_tl {#1}
\regex_replace_all:nnN
{ (\D+) (\d+(\.\d*)) }
{ \c{uiy_do:nn} \cB{\1\cE} \cB{\2\cE} }
\l_uiy_result_tl
\tl_use:N \l_uiy_result_tl
}
\cs_new_protected:Npn \uiy_do:nn #1#2 { \use:c {#1} {#2} }
\ExplSyntaxOff


This time, I catch both the sequence of non-digits, and the number, as captured groups, \1 and \2. Each such occurrence is replaced by the macro \uiy_do:nn (the \c escape in this case indicates "build a comman"), then a begin-group (\cB) character { (this time, \c indicates the category code), then the non-digits (\1), then an end-group (\cE) character }, then another \cB{, the number, and a closing \cE}.

After that, the token list looks like \uiy_do:nn {ljksadflh} {1}. We then simply use its contents with \tl_use:N. The final step is to actually define \uiy_do:nn. Here, I defined it as simply building a command from #1, and giving it the argument #2. This very simple action could be done at the replacement step using \c{\1} for "build a command from the contents of group \1", and technically it would be slightly better, producing an "undefined control sequence" error if the relevant command is not defined. Another option for that error detection to happen is to replace \use:c {#1} {#2} by \cs_if_exist_use:cF {#1} { \msg_error:nnx { uiy } { undefined-command } } {#2}, with an appropriately defined error message.

-
This taste lie black forest cake:) Thanks for the regex. Great package. – Yiannis Lazarides Mar 17 '12 at 2:30
I like this method as it is much easier to extend by just changing the regexp. I see that latex3 has the ability to map tokens to functions... well, only one function. Is it possible to map a token list to a "function list". Basically I would like to extend your macro to map non-numbers to macros. Basically my strings are <macro><value> pairs. – Uiy Mar 17 '12 at 5:18
@Uiy: So you want to replace nondigits123.456moreletters34.56 by \nondigits{123.456}\moreletters{34.56}? – Bruno Le Floch Mar 17 '12 at 5:20
@YiannisLazarides Thanks. That (and some string conversion functions) was 4 months of my life :). – Bruno Le Floch Mar 17 '12 at 5:21
@BrunoLeFloch This is the good part of life, enjoy it! Better than working on some obscure thesis:) – Yiannis Lazarides Mar 17 '12 at 6:54

This one works purely by expansion, so is safe in write etc:

\$ tex split
This is TeX, Version 3.1415926 (Web2C 2010)
(./split.tex
)
No pages of output.
Transcript written on split.log.

\def\xytest#1{?%
\ifnum#1<46?\else
\ifnum#1>57?\else
\ifnum#1=47?\else!\fi\fi\fi}

\def\x#1{%
\ifx\relax#1%
\else
\if\xytest#1%
#1\expandafter\expandafter\expandafter\xx
\else
#1\expandafter\expandafter\expandafter\y
\fi
\fi}

\def\xx#1{%
\ifx\relax#1%
\else
\if\xytest#1%
#1\expandafter\expandafter\expandafter\xx
\else
, #1\expandafter\expandafter\expandafter\y
\fi
\fi}

\def\y#1{%
\ifx\relax#1%
\else
\if\xytest#1%
, #1\expandafter\expandafter\expandafter\xx
\else
#1\expandafter\expandafter\expandafter\y
\fi
\fi}

\bye

-
David's raw tricks are a tutorial class for learners. No prefab, hooded macros. – Ahmed Musa Mar 17 '12 at 14:45
\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
% Document commands
% \NewDocumentCommand\parsestring { m } { \uiy_parse_string:n { #1 } }

\NewDocumentCommand\MyMacro { s O{0} m }
{
\IfBooleanTF { #1 }
{
\uiy_parse_string:n { #3 }
\uiy_my_macro_parsed:n { #2 }
}
{
\clist_set:Nn \l_uiy_parsed_string_clist { #3 }
\uiy_my_macro_parsed:n { #2 }
}
}

% Inner commands
\tl_const:Nn \c_uiy_numbers_tl {0123456789.}
\tl_new:N \l_uiy_parsed_string_tl
\clist_new:N \l_uiy_parsed_string_clist
\seq_new:N \l_uiy_main_seq
\bool_new:N \l_uiy_previous_is_number_bool
\cs_generate_variant:Nn \tl_if_in:NnTF {NV}

\cs_new:Npn \uiy_parse_string:n #1
{
\tl_clear:N \l_uiy_parsed_string_tl
\seq_set_split:Nnn \l_uiy_main_seq {} { #1 }
\seq_pop_left:NN \l_uiy_main_seq \l_uiy_parsed_string_tl
\tl_if_in:NVTF \c_uiy_numbers_tl \l_uiy_parsed_string_tl
{ \bool_set_true:N \l_uiy_previous_is_number_bool }
{ \bool_set_false:N \l_uiy_previous_is_number_bool }
\seq_map_inline:Nn \l_uiy_main_seq
{
\tl_if_in:NnTF \c_uiy_numbers_tl { ##1 }
{
\bool_if:NF \l_uiy_previous_is_number_bool
{ \tl_put_right:Nn \l_uiy_parsed_string_tl { , } }
\tl_put_right:Nn \l_uiy_parsed_string_tl { ##1 }
\bool_set_true:N \l_uiy_previous_is_number_bool
}
{
\bool_if:NT \l_uiy_previous_is_number_bool
{ \tl_put_right:Nn \l_uiy_parsed_string_tl { , } }
\tl_put_right:Nn \l_uiy_parsed_string_tl { ##1 }
\bool_set_false:N \l_uiy_previous_is_number_bool
}
}
\clist_set:NV \l_uiy_parsed_string_clist \l_uiy_parsed_string_tl
}

\cs_new:Npn \uiy_my_macro_parsed:n #1
{
\int_compare:nTF { #1 = 0 }
{
\clist_map_inline:Nn \l_uiy_parsed_string_clist
{
% here you do something with the successive items
item: ~  ##1 \par
}
}
{
% here you do something with the #1-th item
\clist_item:Nn \l_uiy_parsed_string_clist { #1 - 1 } \par
}
}
\ExplSyntaxOff

\begin{document}

\bigskip

\bigskip

\bigskip

\bigskip

\end{document}


With this you see that an unparsed string is called with the *-variant; the optional argument tells to access only one item of the list. Of course it's not possible to say more without knowing the intended usage of the data.

-
can you show me how to pass this comma dileminated string as an array to a macro? So \MyMacro{\parsestring{ljksadflh23898129hfafh0324.22234}} will be the same as \MyMacro{ljksadflh,23898129,hfafh,0324.22234}? – Uiy Mar 16 '12 at 19:50
Also, is there a way to simply get at the elements. I want to access elements in the list analogous to a array in most procedural languages. e.g., myarray[0] = ljksadflh, myarray[1] = 23898129, etc... – Uiy Mar 16 '12 at 20:00
1. It all depends on what your \MyMacro is supposed to do; if you want to pass it the unparsed string, then define it in a suitable way (I can't say how, of course, since I don't know what it is to be doing). 2. Yes, when the parsed list is in a token list variable it's very easy to get at the n-th item. But in this case it would be better to make it into a "clist" variable. – egreg Mar 16 '12 at 20:29
It doesn't matter what \MyMacro is doing. I simply want to pass each element of the list you created to it so that #1 = ljksadflh, #2 = 23898129, etc... – Uiy Mar 16 '12 at 20:32
I simply want \MyMacro{\parsestring{ljksadflh23898129hfafh0324.22234}} to be functionally equivalent to \MyMacro{ljksadflh,23898129,hfafh,0324.22234} – Uiy Mar 16 '12 at 20:33

Here is my own latex3 version similar to egreg's. It is an outlined version but can be modified. Posting here just for reference. It's a little more clean in some ways as it uses some direct parsing on the tokens instead of creating a token list then parsing the token list. (it doesn't use any seq lists since they are not needed)

\ExplSyntaxOn
\tl_const:Nn \tlc_Digits {0123456789.}
\tl_const:Nn \tlc_Space { }
\bool_new:N \b_EON
\bool_new:N \b_FirstDigit

\NewDocumentCommand\ParseCmd{m}
{\tl_if_blank:nTF {#1} {}{

% Initialize Variables
\bool_set_false:N \b_EON
\bool_set_true:N \b_FirstDigit

% Parse Tokens
\tl_map_inline:nn {#1}
{
% Checks to see if the token is a digit or not
\tl_if_in:NnTF \tlc_Digits {##1}
{
\bool_set_false:N \b_EON

\bool_if:NTF \b_FirstDigit { [ } { , }
##1

\bool_set_false:N \b_FirstDigit
}
{   % Token not a digit

\bool_set_true:N \b_FirstDigit
\tl_if_in:NnTF \tlc_Space {##1} { } { }
\bool_if:NTF \b_EON { } { ], }
\bool_set_true:N \b_EON
}

}

% used for very last token
\bool_if:NTF \b_EON { } { ] }

}}
\ExplSyntaxOff
`
-