# Description

I'm trying to define a macro, say \CountToken, that analyzes a selected text, counts the number of occurrences of a given token in this text and stores the number of occurrences in a counter. The token can also be in the form \<some_macro_name>, where \<some_macro_name> has the function of a separator. For my purposes, only such tokens are important that are not hidden inside a group.

The syntax assumed is

\CountToken[<selected_token>,\<counter_name>]
<text>
\finish


The result should be as follows:

• create the counter \<counter_name>,
• count the number of occurrences of <selected_token> within <text> that is delimited by \finish,
• store the number of occurrences in \<counter_name>.

Example:

\CountToken[i,\cnti]
Lorem ipsum dolor $\frac{21}{3}$ sit amet,
\finish


Here, \the\cnti gives 6.

# Own attempt (MWE)

Here is my minimal working example.

\documentclass{article}

\def\finish{}
\newcount\tmp

\def\CountToken[#1,#2]{%
\def\TestedToken{#1}
\tmp=0
\newcount#2
\let\TotalOccurrence#2
\let\next\TestNext\next}

\long\def\TestNext#1{%
\def\CurrentToken{#1}%
\ifx#1\finish
\def\next{\TotalOccurrence=\the\tmp\relax}
\else
\ifx\CurrentToken\TestedToken
\fi
\fi
\next}

\begin{document}

%-----------------------
\CountToken[i,\cnti]
Lorem ipsum dolor $\frac{21}{3}$ sit amet,
\finish
%-----------------------
number of i's: \the\cnti

%-----------------------
\CountToken[f,\cntf]
Lorem ipsum dolor $\frac{21}{3}$ sit amet,
\finish
%-----------------------
number of f's: \the\cntf

%-----------------------
\CountToken[\something,\cntsomething]
Lorem ipsum dolor $\frac{21}{3}$ sit amet,
\finish
%-----------------------
number of \verb|\something|'s: \the\cntsomething

\end{document}


Everything seems to be running fine. However, changing the value of the fraction to 22/3 generates a compilation error. Experimentally, I found that this happens if the numerator of the fraction starts with two same digits. I know that, e.g., writing \frac{{22}}3 helps, but this is not a serious way for me.

# Request

I would like to know several things.

1. How can I improve my definition of \CountToken such that everything is running fine? (Please, only LaTeX solutions.)
2. Could you describe the origin of the error described above?
3. Are there any further safe ways to count the selected token in a text snippet containing general mathematical stuff?
• you are allocating a new register each use, it's almost never good to have \newcount in a macro. Also you are adding very many space characters each time. You ask for a plain tex solution but plain tex makes it an error to nest \newcount in this way, the definition you show is latex-only. – David Carlisle Jan 24 at 22:51
• Thank you. I corrected my question. – Marian G. Jan 24 at 22:54
• you say Here, \the\cnti gives 6 what would you want the count to be for Lorem \emph{ipsum} dolor $\frac{i}{j}$ sit amet, \textbf{consectetur adipiscing elit}. the code here (and Marcel's answer) would give 1 as it doesn't descend into groups. Is that OK? – David Carlisle Jan 24 at 23:00
• @DavidCarlisle: I greatly appreciate your strict and rigorous comments concerning my original question. I've improved the introduction and mentioined that only the 'outer' tokens are important for me. I think that counting all the tokens (including the hidden ones) is currently beyond the scope of my knowledge. If I have time, I'll think about it. – Marian G. Jan 25 at 21:47

The problem is not math mode but any braced expression where the first two tokens are the same. Take your example of "{22}":

At some point, you looked at everything before {22}. Then the argument #1 of \TestNext becomes 22 and therefore \ifx#1\finish becomes \ifx22\finish.

Here \ifx 22 is true, so \finish is expanded, leading to the error.

You can fix this by comparing \CurrentToken instead of using #1 directly. You also need a helper macro storing your terminator:

\documentclass{article}

\def\finish{}
\def\finishmarker{\finish}
\newcount\tmp

\def\CountToken[#1,#2]{%
\def\TestedToken{#1}%
\tmp=0
\newcount#2%
\let\TotalOccurrence#2%
\let\next\TestNext\next}

\long\def\TestNext#1{%
\def\CurrentToken{#1}%
\ifx\CurrentToken\finishmarker
\def\next{\TotalOccurrence=\the\tmp\relax}%
\else
\ifx\CurrentToken\TestedToken
\fi
\fi
\next}

\begin{document}

%-----------------------
\CountToken[i,\cnti]
Lorem ipsum dolor $\frac{22}{3}$ sit amet,
\finish
%-----------------------
number of i's: \the\cnti

%-----------------------
\CountToken[f,\cntf]
Lorem ipsum dolor $\frac{22}{3}$ sit amet,
\finish
%-----------------------
number of f's: \the\cntf

%-----------------------
\CountToken[\something,\cntsomething]
Lorem ipsum dolor $\frac{22}{3}$ sit amet,
\finish
%-----------------------
number of \verb|\something|'s: \the\cntsomething

\end{document}


Another problem with the code is that it only counts "outer" tokens, meaning that tokens "hidden" inside of {} are ignored.

• \ifdefined#2\else\newcount#2\fi would be marginally better (it doesn't check that #2 is actually a counter if defined; this could be fixed, of course). Also the endlines should be cured. – egreg Jan 24 at 22:57
• Thank you. I need only 'outer' tokens. I'll test your code tomorrow. – Marian G. Jan 24 at 22:58
• @Marcel Krüger: I understand the mechanism now and therefore the origin of the problem. Many thanks for sharing your ideas. – Marian G. Jan 25 at 21:33

You do \def\CurrentToken{#1} but then do \ifx#1\finish instead of \ifx\CurrentToken\finish. Wait! This wouldn't yield true if we're at the end!

Yes, if you do \def\finish{\finish}. Of course you have to be careful that \finish doesn't end up in some place where it's expanded.

The \tmp counter is redundant: you have \TotalOccurrence available, use it.

\documentclass{article}

\def\finish{\finish}

\def\CountToken[#1,#2]{%
\def\TestedToken{#1}%
\ifdefined#2#2=0 \else\newcount#2\fi
\let\TotalOccurrence#2%
\let\next\TestNext\next
}

\long\def\TestNext#1{%
\def\CurrentToken{#1}%
\ifx\CurrentToken\finish
\let\next\relax
\else
\ifx\CurrentToken\TestedToken
\fi
\fi
\next
}

\begin{document}

%-----------------------
\CountToken[i,\cnti]
Lorem ipsum dolor $\frac{21}{3}$ sit amet,
\finish
%-----------------------
number of i's: \the\cnti

%-----------------------
\CountToken[f,\cntf]
Lorem ipsum dolor $\frac{21}{3}$ sit amet,
\finish
%-----------------------
number of f's: \the\cntf

%-----------------------
\CountToken[\something,\cntsomething]
Lorem ipsum dolor $\frac{21}{3}$ sit amet,
\finish
%-----------------------
number of \verb|\something|'s: \the\cntsomething

\end{document}


Just for completeness, here's an expl3 version:

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\NewDocumentCommand{\CountToken}{mm+m}
{% #1 = token to check, #2 = counter, #3 = text to count in
\int_zero_new:N #2
\tl_map_inline:nn { #3 }
{
\tl_if_eq:nnT { #1 } { ##1 } { \int_incr:N #2 }
}
}
\ExplSyntaxOff

\begin{document}

%-----------------------
\CountToken{i}{\cnti}{
Lorem ipsum dolor $\frac{21}{3}$ sit amet,
}
%-----------------------
number of i's: \the\cnti

%-----------------------
\CountToken{f}{\cntf}{
Lorem ipsum dolor $\frac{21}{3}$ sit amet,
}
%-----------------------
number of f's: \the\cntf

%-----------------------
\CountToken{\something}{\cntsomething}{
Lorem ipsum dolor $\frac{21}{3}$ sit amet,
}
%-----------------------
number of \verb|\something|'s: \the\cntsomething

\end{document}


A similar problem was solved in https://tex.stackexchange.com/a/525246/4427 (also counting tokens or sets of tokens inside braces).

Let's look at the code in more detail.

The macro \CountToken is defined to have three mandatory arguments, the last of which can contain also \par tokens (because of the + prefix).

The command first sets the counter to zero (argument #2) or allocates a new one if not yet existing (the effect is similar to \ifdefined#2#2=0 \else\newcount#2\fi).

Next we map the third argument: a token list can be seen as a list of items (spaces between items are ignored, though), where braced groups count as a single item. Each item is passed to the code specified as the second argument, in this case

\tl_if_eq:nnT { #1 } { ##1 } { \int_incr:N #2 }


Here ##1 stands for the current item in the mapping. The T stands for True: the third argument is only executed if the two token lists given as the first two arguments are equal.

It's essentially the same code as above, but the details of building the loop and checking equality are already provided by standard functions.

• Thank you for your time. I am only be able to understand the first very instructive part of your answer. Although the expl3 notation is very short, it is completely strange to me. I'll hope to learn from your second approach sometime in the future. – Marian G. Jan 25 at 22:00
• @MarianG. I added some details. You find descriptions of expl3 by doing texdoc interface3. – egreg Jan 25 at 22:13
• Very helpful. Thanks. – Marian G. Jan 25 at 22:20