# Parsing strings containing diacritical marks (macros?)

I apologize that this question may seem very much like several other recent ones I have asked (Parsing leading hardspaces, Parsing a \$as part of an improved \getargs command). They are all related to using a recursive TeXnique to parse strings. David Carlisle has already helped a great deal in the prior queries in pointing me to \protected@edef to handle special (e.g., \$) characters and in using \let to redefine hardspaces to something innocuous (either nulls or soft spaces).

But now I find the algorithm chokes on diacritical marks like umlauts (\"u). I guessing that these are not "characters" but expandable "macros" which might explain why an algorithm that searches character by character may choke. So I'm looking for a clever workaround, something like a temporary \let or \catcode redefinition that would allow the algorithm to digest these types of "characters".

So the code is below. I commented out the one line (2nd from end) which will cause the algorithm to choke. What the algorithm does is looks at the argument and parses it into single words. It places the first word in \argi, the second in \argii, etc. and the total number of words in \narg. And its much faster than its predecessor in the stringstrings package. It works great with alphanumeric inputs, special characters, hard and soft spaces, but not (yet) with diacritical marks.

\documentclass{article}
\makeatletter
\usepackage{ifnextok}
\usepackage{ifthen}
\def\string@end{$} \def\converttilde{T} \newcounter{arg@index} \let\SaveHardspace~ \def\getargsF#1{% \if T\converttilde\def~{ }\fi% \protected@edef\the@string{#1}% \def\argi{}\setcounter{arg@index}{1}% \expandafter\parse@Block\the@string\string@end% \let~\SaveHardspace% } %ifthenelse needed when leading spaces end up as \argi \def\parse@Block{\IfNextToken\string@end% {\edef\narg{\arabic{arg@index}}\@gobble}% {\IfNextToken\@sptoken{\addtocounter{arg@index}{1}% \ifthenelse{\equal{\argi}{}}{\addtocounter{arg@index}{-1}}{}% \expandafter\def\csname arg\roman{arg@index}\endcsname{}% \add@to{\parse@Block}}% {\add@to{\parse@Block}}}} \def\add@to#1#2{\if\@sptoken#2\else\expandafter\protected@edef% \csname arg\roman{arg@index}\endcsname% {\csname arg\roman{arg@index}\endcsname#2}\fi#1} \makeatother \begin{document} \getargsF{mein hut} \argi~\argii$<$NO PROBLEM\\ BUT PUT UMLAUT ON THE u AND IT BREAKS %\getargsF{mein h\"ut} \argi~\argii \end{document}  - The answers by Scott H, unbonpetit, and David Carlisle are all excellent responses. I don't know xparse, but it looks amazingly powerful. The ability of unbonpetit's approach to dissect all the macros is extremely noteworthy, and I have to still unravel the magic performed by Mr. Carlisle (though for my larger application I think I will have to step through token by token). It will take me a while to digest them to see which works best with my intended direction. Thanks all. – Steven B. Segletes Mar 9 '13 at 14:02 ## 3 Answers Once you have expanded everything in the original input with \protected@edef you really don't need to step through token by token which makes it hard to handle any kind of brace group or command taking arguments. This just scans with a delimited argument looking for spaces which is much simpler.  \documentclass{article} \makeatletter \def\string@end{$\SaveHardspace}
\def\converttilde{T}
\newcounter{arg@index}
\let\SaveHardspace~%%%
\def\getargsF#1{%
\if T\converttilde\def~{ }\fi
\protected@edef\the@string{#1}%
\setcounter{arg@index}{0}%
\lowercase{\expandafter\parse@Block\the@string} \string@end
\let~\SaveHardspace
}

\def\parse@Block#1 {%
\stepcounter{arg@index}%
\@namedef{arg\roman{arg@index}}{#1}%
\futurelet\tmp\parse@Block@}

\def\parse@Block@{%
\ifx\tmp\string@end\edef\narg{\thearg@index}\expandafter\@gobble
\else\expandafter\parse@Block\fi}

\makeatother
\begin{document}
\getargsF{mein hut} \argi~\argii~[\narg] $<$ NO PROBLEM

BUT PUT UMLAUT ON THE u AND IT BREAKS

\getargsF{mein h\"ut see}  \argi~\argii~[\narg]
\end{document}

-
The use of \futurelet looks like a really useful command for these sorts of problems. Also, I see \@namedef is a good typing saver. – Steven B. Segletes Mar 10 '13 at 0:23
1) Is there significance to the triple %%% following the \let? 2) What's the significance of the \lowercase? 3) Why did you add a \SaveHardspace to the \string@end definition? 4) What does putting \narg in brackets [] do? – Steven B. Segletes Mar 10 '13 at 2:23
@unbonpetit It is difficult to select the Accepted Answer on this question, since all answers were great! unbonpetit's coding is prettiest, and perhaps should be adopted by others following this track (and I recommend others to "like" it as useful). But my request was for a "workaround" indicating minimum changes from what I started with. With that as the metric, the xparse solution gets eliminated while David Carlisle's code accomplished the goal with the minimum disruption to my original MWE. Thus, he gets his answer accepted on technical grounds. My thanks to all. – Steven B. Segletes Mar 10 '13 at 2:35

The following works with the input described above.

\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
\NewDocumentCommand{\getargs}{m}{
\get_args:n {#1}
}
\int_new:N \narg
\cs_new_protected:Npn \get_args:n #1 {
\seq_set_split:Nnn \l_tmpa_seq {~} {#1}
\seq_map_inline:Nn \l_tmpa_seq {
\int_incr:N \l_tmpa_int
\tl_set:cx {arg\int_to_roman:n {\l_tmpa_int}}{##1}
}
\int_set:Nn \narg {\l_tmpa_int}
\int_zero:N \l_tmpa_int
}
\ExplSyntaxOff
\begin{document}
\getargs{mein hut} \argi~\argii $<$NO PROBLEM\\
BUT PUT UMLAUT ON THE u AND IT BREAKS
\getargs{mein h\"ut} \argi~\argii
\end{document}

-

Here is a solution with no expansion, able to parse macros and groups {...}:

\documentclass{article}
\makeatletter
\newcount\arg@index
\newtoks\code@toks

\def\getargsF#1{%
\arg@index=1
\code@toks{}%
\parse@i#1\parse@stop
}

\def\parse@stop{\parse@stop}

\def\parse@i{\futurelet\nxttok\parse@ii}

\def\parse@ii{%
\ifx\nxttok\parse@stop \let\next@action\parse@stop@i
\else
\else
\else \let\next@action\testtoken
\fi
\fi
\fi
\next@action
}

\def\parse@stop@i\parse@stop{\assign@arg}

\assign@arg
\code@toks{}%
\parse@i
}

\parse@i
}

\def\testtoken#1{%
\if\noexpand~\noexpand#1%
\ifnum\catcode#1=\active
\assign@arg
\code@toks{}%
\else
\fi
\else
\fi
\parse@i
}

\def\assign@arg{%
\expandafter\edef\csname arg\romannumeral\arg@index\endcsname{\the\code@toks}%
}
\makeatother
\begin{document}
\getargsF{abcd efgh~h\"ut}\argi\#\argii\#\argiii\#

\getargsF{maths $1+1=2$ \textbf{b\textit{ol}d} \^a\'i\'o~hardspc~end}\argi\#\argii\#\argiii\#\argiv\#\argv\#\argvi\#

\catcode\~12
\getargsF{maths $1+1=2$ \textbf{b\textit{ol}d} \^a\'i\'o~hardspc~end}\argi\#\argii\#\argiii\#\argiv\#
\end{document}


which gives

-
Just as \protected@edef was a major revelation to me the other week, so your answer reveals the equally amazing \code@toks command. If there are any documents that discuss this command, I'd love a pointer. – Steven B. Segletes Mar 9 '13 at 23:36
Also, \futurelet is new to me, but a very useful command, I see in Will Robertson's answer at tex.stackexchange.com/questions/850/… – Steven B. Segletes Mar 10 '13 at 0:14
@StevenB.Segletes \code@toks isn't a pre-defined command it is just a register defined on line 4, same as a count register but for token lists rather than numbers. \futurelet is the tex primitive behind latex's \@ifnextchar or the \IfNextToken that you used. – David Carlisle Mar 10 '13 at 0:42