# Basics of parsing

I'm trying to master the art of parsing text one character at a time. The following is what I've discovered so far. Is this as good as it gets? I would really prefer not to do all those \edefs as they waste RAM.

\documentclass{article}
\tracingmacros=1

\newcommand{\scanA}[1]% #1 = text to scan
{\futurelet\token\scanB#1 % does nothing for me
\scanC#1 % splits off first token
\edef\next{#1\relax}%
\expandafter\scanD\next% refined version
\expandafter\scanD\next% okay, scaning underway
\expandafter\scanD\next
\expandafter\scanE\next{} % end
}

\def\scanB #1 {\noindent scanB:\par token \token \par\#1 #1 \par}

\def\scanC #1#2 {\noindent scanC:\par\#1 #1 \par\#2 #2 \par}

\def\scanD #1#2\relax{\noindent scanD:\par\#1 #1 \par\#2 #2 \par
\edef\next{#2\relax}}

\def\scanE #1#2\relax{\noindent scanE:\par\#1 #1 \par\#2 \meaning #2
\ifx#2\relax \relax\par The end.\par\fi}

\begin{document}
\scanA{test}
\end{document}


Lessons Learned: (Okay, I should have known some of this already, but didn't.)

\documentclass{article}
\tracingmacros=1

\def\scanA#1{(#1)}% copies 1 token
\def\scanB#1 {(#1)}% copies 1 word

\newcommand{\scan}[1]{\scanC#1\END}% loop until \END
\def\scanC#1{\ifx#1\END\else(#1)\expandafter\scanC\fi}% expand \fi before \scanC

\newcommand{\scanwords}[1]{\let\between=\empty\scanD#1 \END}
\def\scanD#1 {\ifx#1\END\else\between
\let\between=\wordfill% insert \wordfill between words
\scanC#1\END% scan letters of word
\expandafter\scanD\fi% expand \fi before \scanD
}
\def\END{almost anything}
\def\wordfill{( )}% \def once, \let repeatedly

\begin{document}
\Huge
\scanA two words

\scanB two words

\scan{t e s t}% ignores spaces

\scanwords{two words}
\end{document}


All of the answers are better than my first attempt, but I need to accept one to "close" the question. Since my goal was to LEARN how to scan better...

Here is another variant I've been playing with. It stores the text as a token list.

\documentclass{article}
\usepackage{lipsum}

\def\END{\END}

\newtoks\mytoks

\def\parse{\futurelet\next\special}% some tokens are ignored
\def\special{% \space and \bgroup
\expandafter\ifx\space\next\relax \mytoks=\expandafter{\the\mytoks\space}\fi%
\ifx\bgroup\next\relax \expandafter\copygroup\else
\expandafter\normal\fi}
\def\copygroup#1{\mytoks=\expandafter{\the\mytoks{#1}}\parse}%
\def\normal#1{%
\ifx\END#1\relax
\the\mytoks% end of environment
\else
\mytoks=\expandafter{\the\mytoks#1}%
\expandafter\parse\fi}

\begin{document}
\parse
\begin{center}
enviroment test
\end{center}
\noindent\hbox{parse} test
\END
\end{document}

• If you're doing this for an actual purpose and not just for learning, you should check out expl3 if you haven't already. That said, I'm not sure exactly what you're trying to dowhat is as good as it gets? I'm not sure what you're referring to. – Sean Allred Mar 14 '15 at 18:18
• Besides acquiring a new skill, I would like to change colors every time I hit a catcode 7 (I think that's the right one) in order to answer tex.stackexchange.com/questions/232990/two-coloured-hyphenation without having to learn a new language (eg. LuaLaTeX or expl3). – John Kormylo Mar 14 '15 at 18:43
• Out of topic: How did you generate random question sheets as in here? – Kim Jong Un Jul 9 '17 at 17:23

The accepted TeX solution here includes several problems. One of them is that whole argument is read at each step, no only one token. The second is that the recursive loop in the accepted code generates the nested \if...\fi construction which is very limited in TeX.

So, I show here the common scanner declared by TeX primitives without the problems described above. The scanning of the spaces is allowed, but braces are not allowed (for simplicity).

\def\scan#1{\scanA#1\end}
\def\scanA{\futurelet\next\scanB}
\def\scanB{\expandafter\ifx\space\next \expandafter\scanC \else \expandafter\scanE \fi}
\def\scanC{\afterassignment\scanD \let\next= }
\def\scanD{\scanE{ }}
\def\scanE#1{\ifx\end#1\else
(#1)% <- The processing over one token is here
\expandafter \scanA \fi
}

\scan{abcdef ghijkl mno}

\bye


Edit: If you leave the space behavior unchanged (i.e. they are ignored), then the code is much more simple:

\def\scan#1{\scanA#1\end}
\def\scanA#1{\ifx\end#1\else
(#1)% <- The processing over one token is here
\expandafter \scanA \fi
}

• Interesting. I looked up \afterassignment (knuth, p279), but the use of blanks in \scanC and \scanD could use some explaining. – John Kormylo Mar 15 '15 at 15:44
• ...explaining: \let\next=<one-space> consumes the detected space token from input stream and runs \scanD after this assignment. And the \scanD runs \scanE{ }, i.e the \scanE macro with explicit space token as the parameter. – wipet Mar 15 '15 at 20:56
• I eventually realized it was all related to detecting spaces. Without spaces you can really simplify everything, but with spaces adding or removing almost anything crashes. About the only change I could make was to \def\scanD{( )\expandafter\scanA} (we already know it's not the end). – John Kormylo Mar 15 '15 at 22:10
• Also, you realize that \end = macro:#1->\csname end#1\endcsname \@checkend {#1}\expandafter \end- group \if@endpe \@doendpe \ \if@ignore \@ignorefalse \ignorespaces \ and could conceivably occur in the text. \eof would be the prefect choice, if it were defined. – John Kormylo Mar 15 '15 at 22:18
• The \end is primitive in normal formats of TeX but it is macro in LaTeX. Both variants of the meaning of the \end are acceptable by the test \ifx\end#1 because the \end has very special meaning in both variants. – wipet Mar 16 '15 at 1:38

Scanning one token at a time requires at least distinguishing whether the scanned token is a space or a left brace. This is because you can't remove the scanned token with a one parameter macro in those cases.

First of all, let's see what \futurelet does; your \futurelet\token\scanB tells TeX to look at what token follows \scanB, without removing it, then making a \let\token=<scanned token> assignment and finally “seeing” \scanB, which should make decisions based on the value of \token.

For terminating the scanning, you have to place some special token at the end; this token is frequently a “quark”, say

\def\quark{\quark}


so \scanB can do \ifx\token\quark and, in this case, stop the recursion. Let's put into play what we have till now:

\makeatletter
\def\scan@quark{\scan@quark}% if we find it in bad places, we'll know!
\newcommand\scan[1]{\futurelet\@let@token\scan@aux@i#1\scan@quark}
\def\scan@aux@i{%
\ifx\@let@token\scan@quark
\expandafter\@gobbletwo
\else
\expandafter\@firstofone
\fi
{\scan@aux@ii}%
}


The macro \scan@aux@ii should now go on with other tests. I used \@gobbletwo in the “true” case so to gobble \scan@aux@ii and \scan@quark.

If instead you want just to split the input at a certain token, a better approach is using delimited arguments: you can find several examples on the site. With expl3 it's quite easy, because there are built in functions that do the job.

So, say you have an input such as \word{abc^def^ghi} that you want to print with alternating colors. Here's an implementation:

\documentclass{article}
\usepackage{xparse,xcolor}

\ExplSyntaxOn
\NewDocumentCommand{\word}{m}
{
\kormylo_word:n { #1 }
}

\seq_new:N \l_kormylo_word_fragment_seq
\bool_new:N \l_kormylo_second_color_bool

\cs_new_protected:Npn \kormylo_word:n #1
{
\kormylo_change_color:
\seq_set_split:Nnn \l_kormylo_word_fragment_seq { ^ } { #1 }
\seq_use:Nn \l_kormylo_word_fragment_seq { \kormylo_change_color: }
}

\cs_new_protected:Npn \kormylo_change_color:
{
\bool_if:NTF \l_kormylo_second_color_bool
{ \color{second} \bool_set_false:N \l_kormylo_second_color_bool }
{ \color{first} \bool_set_true:N \l_kormylo_second_color_bool }
}
\ExplSyntaxOff

\colorlet{first}{black}
\colorlet{second}{red}

\begin{document}

\word{su^per^cal^i^frag^i^lis^tic^ex^pi^al^i^do^cious}

\end{document}


Note that you can use spaces around the separator token for better input, such spaces will be disregarded.

The macros could be extended to allow spaces in the input: just split at spaces and do a mapping.

\documentclass{article}
\usepackage{xparse,xcolor}

\ExplSyntaxOn
\NewDocumentCommand{\words}{m}
{
\kormylo_words:n { #1 }
}

\seq_new:N \l_kormylo_word_seq
\seq_new:N \l_kormylo_word_fragment_seq
\bool_new:N \l_kormylo_second_color_bool

\cs_new_protected:Npn \kormylo_words:n #1
{
\seq_set_split:Nnn \l_kormylo_word_seq { ~ } { #1 }
\seq_map_inline:Nn \l_kormylo_word_seq
{
\kormylo_word:n { ##1 }
\c_space_tl
}
}

\cs_new_protected:Npn \kormylo_word:n #1
{
\kormylo_change_color:
\seq_set_split:Nnn \l_kormylo_word_fragment_seq { ^ } { #1 }
\seq_use:Nn \l_kormylo_word_fragment_seq { \kormylo_change_color: }
}

\cs_new_protected:Npn \kormylo_change_color:
{
\bool_if:NTF \l_kormylo_second_color_bool
{ \color{second} \bool_set_false:N \l_kormylo_second_color_bool }
{ \color{first} \bool_set_true:N \l_kormylo_second_color_bool }
}
\ExplSyntaxOff

\colorlet{first}{black}
\colorlet{second}{red}

\begin{document}

\words{su^per^cal^i^frag^i^lis^tic^ex^pi^al^i^do^cious syl^la^ble con^cate^na^tion}

\end{document}


• In the last example, spaces change color as well. I think, with your approach, it is not so easy, to get \words{a^b b b^a ^b^ a ^b} right. – mhchem Mar 14 '15 at 21:04
• @mhchem It would be easy starting all words with black color, if desired. – egreg Mar 14 '15 at 21:23
• I did not mean that. I meant a space should not change color at all. But I was wrong in my accusation. One only needs to remove \kormylo_change_color: from \kormylo_word:n. – mhchem Mar 14 '15 at 21:46
• @egreg You coul use the active approach: \group_begin: \char_set_active_eq:NN \^ \kormylo_change_color: and then call an auxiliar which grabs the argument and ends the group. – Manuel Mar 15 '15 at 12:24
• @Manuel I wouldn't. – egreg Mar 15 '15 at 13:27

For the first answers, I assume that you want to scan pure text, without any groups commands etc. and without any spaces.

This is the (mainly) TeX solution.

\documentclass{minimal}
\begin{document}
\def\END{}
\def\ENDEND{}
\newcommand*\scan[1]{\scani #1\END\ENDEND}
\def\scani#1#2\ENDEND{%
\ifx\END#1%
\else%
(#1)%
\scani#2\ENDEND%
\fi
}
\scan{test}
\end{document}


And here is the LaTeX3 version.

\documentclass{minimal}
\usepackage{expl3}
\begin{document}
\ExplSyntaxOn
\newcommand*\scan[1]
{
\tl_map_inline:nn {#1} { (##1) }
}
\ExplSyntaxOff
\scan{test}
\end{document}


Both will output

(t)(e)(s)(t)

Dealing with spaces, it somewhat tricky. I have a TeX solution here (from Usenet times), but do not understand it myself.

For LaTeX3, here are solutions that can cope with spaces: LaTeX3: tl_map with spaces

Or you use my version, which is

\documentclass{minimal}
\usepackage{expl3}
\begin{document}
\ExplSyntaxOn
\newcommand*\scan[1]
{
\__scanloop: #1 \q_recursion_stop
}
\cs_new:Nn \__scanloop:
{
\peek_meaning_remove:NTF \q_recursion_stop
{}
{
\peek_charcode_remove:NTF \c_space_token
{
(~)
\__scanloop:
}
% else
{
\__scanloop_aux:
}}
}
\cs_new:Npn \__scanloop_aux: #1
{
( #1 )
\__scanloop:
}
\ExplSyntaxOff
\scan{test with spaces}
\end{document}


which will output

(t)(e)(s)(t)( )(w)(i)(t)(h)( )(s)(p)(a)(c)(e)(s)

• I figured to scan for words (spaces) using a separate loop. Also, I had problems with recursive expansion (macros which "call" themselves) but will look into them again. – John Kormylo Mar 15 '15 at 13:39
• To your TeX solution: do you know that your test \ifx\END#1% is equivalent to \ifx\empty#1%, i.e. you needn't to use the special sequence \END? And if the scanned text includes the macro defines as empty then your scanner terminates here. – wipet Mar 15 '15 at 14:54

This is not an answer to the original scanning question. It is an 'optimization' of egreg's solution for the coloring problem. It saves 17 lines of code by use of l3regex.

\documentclass{minimal}
\usepackage{xparse,l3regex,xcolor}

\ExplSyntaxOn
\NewDocumentCommand{\words}{m}
{
\tl_set:Nn \l_tmpa_tl {#1}
\regex_replace_all:nnN { \cU\^ } { \c{kormylo_change_color:} } \l_tmpa_tl  % replace ^
\regex_replace_all:nnN { \s } { \cS_ \c{kormylo_change_color:} } \l_tmpa_tl  % replace spaces
\kormylo_change_color:
\tl_use:N \l_tmpa_tl
}
\bool_new:N \l_kormylo_second_color_bool
\cs_new_protected:Npn \kormylo_change_color:
{
\bool_if:NTF \l_kormylo_second_color_bool
{ \color{second} \bool_set_false:N \l_kormylo_second_color_bool }
{ \color{first} \bool_set_true:N \l_kormylo_second_color_bool }
}
\ExplSyntaxOff

\colorlet{first}{black}
\colorlet{second}{red}

\begin{document}

\words{su^per^cal^i^frag^i^lis^tic^ex^pi^al^i^do^cious syl^la^ble con^cate^na^tion}

\end{document}


Right now, it can scan up to a paragraph, but with an added layer of macro, I can scan multiple paragraphs.

\documentclass{article}
\usepackage{xcolor}
\def\mycolor{1}
\newcommand\myparse[1]{%
\myparsehelpA#1 \relax\relax}
\def\myparsehelpA#1 #2\relax{%
\myparsehelpB#1\relax\relax%
\ifx\relax#2\else\ \myparsehelpA#2\relax\fi
}
\def\myparsehelpB#1#2\relax{%
\ifcat^#1%
\if1\mycolor\def\mycolor{2}\color{red}%
\else\def\mycolor{1}\color{black}%
\fi%
\else#1\fi%
\ifx\relax#2\else
\myparsehelpB#2\relax%
\fi
}
\begin{document}
\myparse{Th^is^ is ^a te^st of the e^merge^ncy broad^cast sy^stem}
\end{document}


The \myparse macro can handle some macros in its argument, but only if they take no arguments themselves. For example, \myparse{Th^is^ is ^a te^st of the e^merge^ncy broad^cast \itshape sy^stem; \bfseries th^is is \mdseries on^ly a test.} yields