# Tokenizing and parsing a word

After a 10+ year LaTeX hiatus I am returning and slowly getting hold again of macro writing.

I now want to parse a string like "fisis" into the first letter "f" and the second half "isis". Case sensitivity is not really an issue.

Valid tokens obey the following rules:

1. The first letter must be one of {a, b, c..., g}
2. The second part of the token consists of exactly 0, 1 or 2 times the token "es" or "is"

For example, "cesis" is invalid as "es" cannot be followed by "is" according to rule 2.

The use case is creating a multi-lingual rendering of the tonality of a musical piece via a macro, e.g.: \tonality{fis}{minor} which would be rendered in German, French and English as follows: {Fis-moll. {\emph Fa dièse mineur.} F sharp minor.}

E.g., the first example "fisis" would be parsed as ( "f"; "is" x 2 ) which would be rendered as "F double sharp" in English (with the \tonalityEN macro) and "Ré double dièse" in French (with the \tonalityFR macro).

How can this be achieved?

-
Welcome to TeX.sx! Usually, we don't put a greeting or a "thank you" in our posts. While this might seem strange at first, it is not a sign of lack of politeness, but rather part of our trying to keep everything very concise. Upvoting is the preferred way here to say "thank you" to users who helped you. –  Benedikt Bauer Jan 5 '13 at 0:04
Is all input assumed to be wellformed, is there supposed to be errorhandling and if yes what do you expect the function to do on malformed input? Also would a luatex solution be acceptable? –  Max Jan 5 '13 at 0:31
Error handling can be basic (e.g., no output) as I think I can improve a basic but working snippet. I am not acquainted with Lua, nor do I know if I have Lua (I have a recent ProTeXt distribution on a Win7 x64 computer), but I can give it a try if I get the proper instructions. –  ShutterFreak Jan 5 '13 at 0:44
On the music front, beware of the different uses of B in different languages: English B is German H; German B is English B flat. –  Andrew Swann Jan 5 '13 at 8:54
Won't this fail in cases of "As", "Asas", "Ases", "Es" and "Eses"? –  cgnieder Jan 5 '13 at 8:57

I don't completely get your question but the following can perform some tests and can be build upon if you can describe how to handle the language switches. I've used the convenient xstring which comes with quite powerful string handling tools as sampled below. First we remove the initial character and then test if the rest of the word has the first two chars of the remaining part. If there is something left out after deleting the detected two-letter it gives an error. After that it's straightforward.

\documentclass{article}
\usepackage{xstring}
\def\parsethis#1{%
\if\relax#1\relax\else%
\StrLeft{#1}{1}[\myfirstletter]
\StrGobbleLeft{#1}{1}[\myrestofword]
\StrLeft{\myrestofword}{2}[\mysample]
\StrCount{\myrestofword}{\mysample}[\numofreps]
\StrDel{\myrestofword}{\mysample}[\remainingchars]
\if\relax\remainingchars\relax%
I found \numofreps\space copies of the string \mysample''.%
\else
There is a mixture of es and is.%
\fi
}

\begin{document}
\parsethis{cisis}\par
\parsethis{feseses}\par
\parsethis{b}\par
\parsethis{basis}
\end{document}


-
Thank you! I managed to build a working solution based on your code! –  ShutterFreak Jan 5 '13 at 2:21
@ShutterFreak My pleasure. Make sure you compile David's code via plain TeX. –  percusse Jan 5 '13 at 2:37
I tried with PdfTeX and with XeTeX and I could not get the example to compile. However I like a lot the expressiveness of the xstring package which I did not know. And that one made solving my problem a piece of cake. I'm happy to re-discover LaTeX! –  ShutterFreak Jan 5 '13 at 2:41

You can possibly check whether the new LaTeX3 programming style suits you:

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\NewDocumentCommand { \tonality } { m m }
{
\ton_tonality:nn { #1 } { #2 }
}

\tl_new:N \l_ton_key_tl
\tl_new:N \l_ton_accidents_tl
\cs_new_protected:Npn \ton_tonality:nn #1 #2
{
% Normalize to uppercase
\tl_to_uppercase:n
{
% Get the key
\tl_set:Nx \l_ton_key_tl { \tl_head:n { #1 } }
% Get the accidents
\tl_set:Nx \l_ton_accidents_tl { \tl_tail:n { #1 } }
}
% check the key
\str_case:onn { \l_ton_key_tl }
{
{A}{ \ton_print_key:n {A} }
{B}{ \ton_print_key:n {B} }
{C}{ \ton_print_key:n {C} }
{D}{ \ton_print_key:n {D} }
{E}{ \ton_print_key:n {E} }
{F}{ \ton_print_key:n {F} }
{G}{ \ton_print_key:n {G} }
}
{ \msg_error:nn { tonality } { bad-key } !! }
% check the accidents
\str_case:onn { \l_ton_accidents_tl }
{
{}    { }
{ES}  { \ton_print_flat: }
{ESES}{ \ton_print_doubleflat: }
{IS}  { \ton_print_sharp: }
{ISIS}{ \ton_print_doublesharp: }
}
{ \msg_error:nn { tonality } { bad-accidents } !! }
\ton_print_mode:n { #2 }
}

\cs_new:Npn \ton_print_mode:n #1 { \nobreakspace #1 }

\cs_new:Npn \ton_print_key:n #1 { #1 }
\cs_new:Npn \ton_print_flat: { \nobreakspace flat }
\cs_new:Npn \ton_print_doubleflat: { \nobreakspace doubleflat }
\cs_new:Npn \ton_print_sharp: { \nobreakspace sharp }
\cs_new:Npn \ton_print_doublesharp: {\nobreakspace doublesharp }

\msg_new:nnnn { tonality } { bad-key }
{ Key~not~in~range~A--G }
{ The~first~character~is~not~good }
\msg_new:nnnn { tonality } { bad-accidents }
{ The~string~for~the~accidents~is~wrong }

\ExplSyntaxOff

\begin{document}
\tonality{A}{minor}

\tonality{fis}{minor}

\tonality{fesis}{major}

\tonality{qisis}{major}
\end{document}

-
I haven't yet ventured into LaTeX3, and I currently don't think I can use LaTeX3 (lilypond-book). But I find your example very interesting. Lots of new constructs that have a lot of use IMHO. Thanks! –  ShutterFreak Jan 5 '13 at 23:38
@ShutterFreak If you add some details about your printing macros, it's possible to add more. Do they depend on the current babel language? –  egreg Jan 5 '13 at 23:45
I don't think I will need babel for this particular macro set, but I will use babel elsewhere in this project (multi-lingual text snippets). Anyway, I will make my LaTeX snippets available to others, I'm now checking with the LilyPond community if they have a space for this sort of snippets (I'm using lilypond-book for combining LilyPond music engraving and LaTeX document typesetting). –  ShutterFreak Jan 5 '13 at 23:49

This example is plain tex but would work in latex of course.

\def\zz#1{%
\edef\tmp{\noexpand\zzformat
\zzz#1\relax\relax\relax\relax\relax\relax}%
%show\tmp
\tmp}

\def\zzz#1#2#3#4#5#6#7{%
{#1}%
\ifx\relax#2{0}{}%
\else\ifx\relax#4{1}{#2#3}%
\else{2}{#2#3}\fi\fi}

\def\zzformat#1#2#3{%
#1 + #2 ( #3 )
}

\zz{a}

\zz{aes}

\zz{fisis}

\bye

-
I more or less see how this works, but sadly the example does not compile on my box. –  ShutterFreak Jan 5 '13 at 1:11
Did you use latex or (as I noted in the answer) tex: the actual code will work in latex but to keep the example short it is plain tex (no \documentclass and ending \bye) –  David Carlisle Jan 5 '13 at 12:46
I used plain TeX indeed (PdfTeX and with XeTeX within TeXworks), as well as LaTeX (PdfLaTeX) for that snippet. It was however instructive in catching up again with TeX argument handling (I had forgotten about the use of \relax for instance). –  ShutterFreak Jan 5 '13 at 22:11
oops sorry there was a \show which isn't an error but a debugging command which may have confused your IDE. I commented it out. the code posted runs without error in pdftex or xetex/tex/luatex (pdftex was used to produce the image) –  David Carlisle Jan 5 '13 at 23:19
It works now, thanks! –  ShutterFreak Jan 5 '13 at 23:39

I managed to find a solution thanks to your input. Below you'll find a documented debug version - I hope it will be useful to others.

\documentclass{article}
\usepackage{xstring}

\def\flatmodifier{es}
\def\sharpmodifier{is}
\def\naturalmodifier{}

\parindent 0cm

\def\parsethis#1{%
{\bf{#1}\par}%
\if\relax#1\relax\else%
\StrLeft{#1}{1}[\myfirstletter]%                            \myfirstletter = root tone
\StrPosition{abcdefg}{\myfirstletter}[\mynote]
\ifnum\mynote=0%
ERROR!!! Invalid note name (\myfirstletter)\par
\else
\StrGobbleLeft{#1}{1}[\myrestofword]%                       Eat first letter
\StrLeft{\myrestofword}{2}[\mysample]%                      \mysample = 2 characters after the root tone
\ifx\mysample\naturalmodifier%                                      (Natural)
\def\mytonemodifier{Natural}%
\else
\ifx\mysample\flatmodifier%                                     (Flat)
\def\mytonemodifier{Flat}%
\else
\ifx\mysample\sharpmodifier%                                (Sharp)
\def\mytonemodifier{Sharp}%
\else
\def\mytonemodifier{\relax}%                    (Invalid)
\fi
\fi
\fi
\if\mytonemodifier\relax
ERROR!!! Invalid note pitch modifier (\mysample)\par
\else
Okay, we can proceed.\par
\StrCount{\myrestofword}{\mysample}[\numofreps]%            \numofreps = # repetitions of \mysample
\StrDel{\myrestofword}{\mysample}[\remainingchars]%         \remainingchars = remaining characters after removing \mysample from the front
\if\relax\remainingchars\relax%                                 IF there are remaining characters
{\small I found \numofreps\space copies of the string \mysample''.\par}%
\ifnum\numofreps<3%
\myfirstletter\ %                                           Display note name
\ifnum\numofreps=0%                                         No modifier
\mytonemodifier\ %                                          Display note modifier
\else
\ifnum\numofreps=2%                                         Double flat/sharp
Double %
\fi
\mytonemodifier\ %                                          Display note modifier
\fi
\else%                                                          ELSE
ERROR: Too many accidental modifiers (\numofreps).%
\fi%                                                            ENDIF
\else
ERROR: There is a mixture of es and is.%
\fi
\fi
\fi
}

\begin{document}
\parsethis{h}\par
\parsethis{b}\par
\parsethis{be}\par
\parsethis{bas}\par
\parsethis{bes}\par
\parsethis{bese}\par
\parsethis{beses}\par
\parsethis{cisis}\par
\parsethis{feseses}\par
\parsethis{fesesis}\par