# Counting tokens

Is it possible to count the tokens/macro parameters. For example:

\def\foo#1{...}
$\foo{\alpha\beta}$ % => \somecounter = 2

-
Sorry, do you mean now tokens OR macro arguments? Both are quite different: A{something}\beta are 13 tokens but only 3 macro arguments. –  Martin Scharrer Feb 3 '11 at 21:31
@Martin Scharrer: I am sorry; I didn't know there is a difference. Tokens is what I mean. But thank you very much for showing a macro parameter counter as well! –  morbusg Feb 3 '11 at 21:44
@morbusg: One thing to note is that when TeX scans for macro arguments it does not simply take the next n tokens, but obeys { } etc. The TeXBook will give you a more details explanation. In short: while { } groups tokens together, it doesn't represent them as one token (but as one macro argument ;-) ). However, other programming languages mostly talk about expressions and allow { } to group multiple expressions together while the group represents one expression by itself. (Sorry if this sounds now very complicated and may not even be 100% correct) –  Martin Scharrer Feb 3 '11 at 23:05
@Martin. What I think you're getting at is that each argument of a macro is a <balanced text>, which means either a single token or a set of tokens wrapped up inside { ... } (assuming normal category codes). –  Joseph Wright Feb 4 '11 at 7:11
@morbusg. To clarify, something like ab{cd}e is 4 items of <balanced text> but 7 tokens. Which value do you actually want? (You might even want 5, ignoring the braces entirely.) –  Joseph Wright Feb 4 '11 at 7:13

Joseph Wright's solution doesn't need the additional helper macros (\@stop and \cleanup@nil):

\makeatletter
\def\counttokens#1{%
\def\counttokens##1{%
\the\numexpr\expandafter\@counttokens##1#1\@nnil\@nnil\relax
}%
}
\counttokens{ }
\long\def\@counttokens#1{%
\csname @\ifx#1\@nnil first\else second\fi oftwo\endcsname
{0\remove@to@nnil}{1+\@counttokens}
}

\def\test{abc de\par}
\counttokens\test
\counttokens{}


And Martin Scharrer's solution can be made more elegant as follows. Amongst other issues, there is no need for the \eatspace macro.

\makeatletter
\def\counttokens#1{%
\def\counttokens##1{%
\let\tokencount\m@ne
\def\@counttokens{\futurelet\@let@token\@@counttokens}%
\expandafter\@counttokens##1#1\@nnil
}%
}
\counttokens{ }
\def\@@counttokens{%
\csname @\ifx\@let@token\@nnil first\else second\fi oftwo\endcsname
\remove@to@nnil{%
\edef\tokencount{\the\numexpr\tokencount+1}%
\afterassignment\@counttokens
\let\@let@token= %
}%
}

\counttokens{A{something}\beta}
\show\tokencount

\counttokens{ 1 2 3 }
\show\tokencount

-
I've edited your answer to mark up the code. For inline snippets, this is done using backticks at each end. For longer blocks, an indent of four spaces is used, or you can use the 010101 button in the editor to mark up an existing block. –  Joseph Wright Feb 4 '11 at 20:02
thanks for posting an improvement on my code. I'm always happy to learn more. –  Martin Scharrer Feb 5 '11 at 1:40
The \expandafter in \expandafter\@counttokens##1#1\@nnil shouldn't be there, should it? It will expand the first token of the argument, which isn't wanted. The code runs well without it. –  Martin Scharrer Feb 5 '11 at 2:06
@Martin: The first token may be the only one and may be a macro to be expanded. Eg, \def\test{abc de\par} \counttokens\test. If the first token isn't expandable (such as plain text or nonactive characters), then \expandafter has no effect. The extra space introduced in the definition of \counttokens was to guide against the submission of empty argument to \counttokens, eg, \counttokens{} or \counttokens\@empty, both of which correctly return 0. –  Ahmed Musa Feb 7 '11 at 21:23
I see, thanks for explaining it. –  Martin Scharrer Feb 7 '11 at 21:33

Depends on whether you want to count spaces. If you don't mind skipping spaces, then it is relatively easy. Using e-TeX, an expandable approach can be generated:

\catcode\@=11\relax
\def\counttokens#1{%
\number\numexpr\expandafter\counttokens@aux@i#1\@stop\@nil0\relax
}
\long\def\counttokens@aux@i#1{%
\ifx#1\@stop
\expandafter\cleanup@nil
\else
1 +
\expandafter\counttokens@aux@i
\fi
}
\long\def\cleanup@nil#1\@nil{}
\def\@stop{\@stop}
\def\test{abc de\par}
\counttokens\test
\bye


(This is essentially the expl3 function \tl_elt_count:N implemented in plain e-TeX.)

An alternative approach is not to require an expandable solution

\newcount\tokencount
\catcode\@=11\relax
\def\:{\let\@sptoken= } \: %
\def\@stop{\@stop}
\def\counttokens#1{%
\begingroup
\tokencount\z@
\expandafter\counttoken@aux@i#1\@stop
}
\def\counttoken@aux@i{%
\futurelet\@let@token\counttoken@aux@ii
}
\def\counttoken@aux@ii{%
\ifx\@let@token\@stop
\expandafter\counttoken@end
\else
\ifx\@let@token\@sptoken
\expandafter\expandafter\expandafter\counttoken@space
\else
\expandafter\expandafter\expandafter\counttoken@std
\fi
\fi
}
\def\:{\counttoken@space} \expandafter\def\: {%
\futurelet\@let@token\counttoken@aux@ii
}
\long\def\counttoken@std#1{%
\counttoken@aux@i
}
\def\counttoken@end\@stop{%
\expandafter\endgroup
\number\tokencount
}
\def\test{abc  de\par}
\counttokens\test
\bye


I've not added any guard code for & tokens in \haligns, which would be needed for robustness in a real solution. (See Appendix D of The TeXbook, for example, to see what I mean.)

-
@Joseph: is it possible to avoid skipping spaces? –  Bruno Le Floch Feb 3 '11 at 21:47
@Bruno. Do you mean in an expandable way? If so, the answer is 'not without LuaTeX', otherwise we'd have it in expl3! –  Joseph Wright Feb 3 '11 at 22:04
@Joseph: I was thinking "non-expandable", but I guess it does not fit with your answer. –  Bruno Le Floch Feb 3 '11 at 22:37
@Bruno. I've modified the answer to give some space counting. You might want to tap the minds of people on c.t.t with this one! –  Joseph Wright Feb 3 '11 at 22:51
@Joseph: "one or more spaces count as one" because \test only contains one space, here (the two spaces are seen as one by TeX's mouth). On the other hand, with \edef\test{abc\space\space de\par}, we get 8 tokens. I still need to understand your solution, but it works! –  Bruno Le Floch Feb 3 '11 at 22:58

Yes, using something like this: (updated version which counts also space-tokens)

\makeatletter
\newcount\tokencount
\def\counttoken#1{%
\tokencount\z@ % set it to zero
\def\x{\futurelet\@let@token\@counttoken}%
\x#1\end@counttoken
}
% Unique endmarker
\def\end@counttoken{\@gobble{end@counttoken}}
% A macro which eats a space, the \@firstofone is required to
% avoid that the space in the parameter text itself is eaten:
\@firstofone{\def\eatspace} {}
% Recursive Macro => Loop
\def\@counttoken{%
\ifx\@let@token\end@counttoken
\let\x\relax
\else
\fi
\ifx\@let@token\@sptoken
% The space token will be eaten
% Note that two spaces will still be combined by TeX!
% If this is not what is wanted the catcode of space
% must be changed beforehand, e.g. using \obeyspaces!
\expandafter\expandafter
\expandafter\x
\expandafter\eatspace
\else
% Eat the next non-space token now, then call \x
\afterassignment\x
\expandafter\let\expandafter\@let@token\expandafter=%
\fi
}


Test:

\counttoken{A{something}\beta}
\showthe\tokencount
% Prints:> 13
\counttoken{123}
\showthe\tokencount
% Prints:> 3


Basically the next token is \let to \@let@token and afterwards (\afterassignment) the token is counted and checked against the endmarker. Then the macro calls itself until the end marker is reached. Counts all tokens except of course the endmarker. You should add some grouping at the correct places, which depend on the exact way you apply the code.

BUT

if you really just want to count macro arguments, i.e. A{something}B would count as 3:

\newcount\argcount
\def\countargs#1{%
\argcount\@z % set it to zero
\@countargs#1\end@countargs
}
% Unique endmarker
%\def\end@countargs{\@gobble{end@countargs}} % doesn't really have to be defined; never executed
\def\end@@countargs{\end@countargs}
% Recursive Macro => Loop
\def\@countargs#1{%
\def\@tempa{#1}%
\ifx\@tempa\@end@@counttoken\else
\expandafter\@countargs
\fi
}

-
in your first code snippet, typos: \@z -> \z@ and \@end@counttoken -> \end@counttoken. Also, spaces are skipped after a \let. –  Bruno Le Floch Feb 3 '11 at 21:45
@Bruno: Thanks for pointing out the typos, I will fix them. I assumed the same behavior from \let as from \futurelet which doesn't skip spaces. That's why \@ifnextchar goes to some trouble to skip them manually. –  Martin Scharrer Feb 3 '11 at 22:33
@Bruno: I know updated the first code to also count space-tokens by using \futurelet instead. See also the comments in the code. –  Martin Scharrer Feb 3 '11 at 22:59
that works indeed! (and I might be able to use it for my question on a token list's head) –  Bruno Le Floch Feb 3 '11 at 23:36

Counting non-space arguments separately:

\documentclass{article}
\usepackage{catoptions}
\makeatletter
\newvariables{count}{}{argcount,spacecount,nonspacecount}[\m@ne]
\def\countargs#1{%
\def\countargs##1{%
\begingroup
\def\end@countargs{\end@countargs}%
\def\countargs@a{\futurelet\next\countargs@c}%
\def\countargs@b####1{\countargs@a}%
\expandafter\countargs@a##1#1\end@countargs
}%
}
\countargs{ }
\def\countargs@c{%
\ifxTF\next\end@countargs{%
\edef\next{%
\endgroup\argcount\the\argcount\relax
\spacecount\the\spacecount\relax
\nonspacecount\the\nonspacecount\relax
}%
\expandafter\next\@gobble
}{%
\ifxTF\next\@sptoken{%
\afterassignment\countargs@a\let\next= %
}{%
}%
}%
}

% Test
\def\reserved@a#1{{#1#1#1A#1{something}#1\beta#1}}
\expandafter\countargs\reserved@a{ }
\edef\result{%
\string\argcount: \the\argcount,\@space
\string\spacecount: \the\spacecount,\@space
\string\nonspacecount: \the\nonspacecount
}
\show\result

\end{document}

-

Martin’s \countargs macro will not count spaces in the argument because leading spaces may be ignored when a macro grabs an argument. If we want to count spaces in the argument (I can’t see a need for that now but it is an interesting scenario), then the following code should work:

\documentclass{article}
\usepackage{catoptions}
\makeatletter
\newvariables{count}{}{argcount,spacecount}[\m@ne]
\def\countargs#1{%
\def\countargs##1{%
\begingroup
\def\end@countargs{\end@countargs}%
\def\countargs@a{\futurelet\next\countargs@c}%
\def\countargs@b####1{\countargs@a}%
\expandafter\countargs@a##1#1\end@countargs
}%
}
\countargs{ }
\def\countargs@c{%
\ifxTF\next\end@countargs{%
\edef\next{%
\endgroup\argcount\the\argcount\relax
\spacecount\the\spacecount\relax
}%
\expandafter\next\@gobble
}{%
\ifxTF\next\@sptoken{%
\afterassignment\countargs@a
\let\next= %
}{%
\countargs@b
}%
}%
}

% Test
\def\reserved@a#1{{#1#1#1A#1{something}#1\beta#1}}
\expandafter\countargs\reserved@a{ }
\edef\result{%
\string\argcount: \the\argcount,\@space
\string\spacecount: \the\spacecount
}
\show\result

\end{document}

-