Counting tokens

Question

Is it possible to count the tokens/macro parameters. For example:

\def\foo#1{...}
$ \foo{\alpha\beta} $ % => \somecounter = 2

Sorry, do you mean now tokens OR macro arguments? Both are quite different: A{something}\beta are 13 tokens but only 3 macro arguments.
@Martin Scharrer: I am sorry; I didn't know there is a difference. Tokens is what I mean. But thank you very much for showing a macro parameter counter as well!
@morbusg: One thing to note is that when TeX scans for macro arguments it does not simply take the next n tokens, but obeys { } etc. The TeXBook will give you a more details explanation. In short: while { } groups tokens together, it doesn't represent them as one token (but as one macro argument ;-) ). However, other programming languages mostly talk about expressions and allow { } to group multiple expressions together while the group represents one expression by itself. (Sorry if this sounds now very complicated and may not even be 100% correct)
@Martin. What I think you're getting at is that each argument of a macro is a <balanced text>, which means either a single token or a set of tokens wrapped up inside { ... } (assuming normal category codes).
@morbusg. To clarify, something like ab{cd}e is 4 items of <balanced text> but 7 tokens. Which value do you actually want? (You might even want 5, ignoring the braces entirely.)

Joseph Wright · Accepted Answer · 2011-02-04 19:38:51Z

Joseph Wright's solution doesn't need the additional helper macros (\@stop and \cleanup@nil):

\makeatletter
\def\counttokens#1{%
  \def\counttokens##1{%
    \the\numexpr\expandafter\@counttokens##1#1\@nnil\@nnil\relax
  }%
}
\counttokens{ }
\long\def\@counttokens#1{%
    \csname @\ifx#1\@nnil first\else second\fi oftwo\endcsname
    {0\remove@to@nnil}{1+\@counttokens}
}

\def\test{abc de\par}
\counttokens\test
\counttokens{}

And Martin Scharrer's solution can be made more elegant as follows. Amongst other issues, there is no need for the \eatspace macro.

\makeatletter
\def\counttokens#1{%
  \def\counttokens##1{%
    \let\tokencount\m@ne
      \def\@counttokens{\futurelet\@let@token\@@counttokens}%
      \expandafter\@counttokens##1#1\@nnil
  }%
}
\counttokens{ }
\def\@@counttokens{%
  \csname @\ifx\@let@token\@nnil first\else second\fi oftwo\endcsname
    \remove@to@nnil{%
        \edef\tokencount{\the\numexpr\tokencount+1}%
    \afterassignment\@counttokens
      \let\@let@token= %
    }%
}

\counttokens{A{something}\beta}
\show\tokencount

\counttokens{ 1 2 3 }
\show\tokencount

I've edited your answer to mark up the code. For inline snippets, this is done using backticks at each end. For longer blocks, an indent of four spaces is used, or you can use the 010101 button in the editor to mark up an existing block.
thanks for posting an improvement on my code. I'm always happy to learn more.
The \expandafter in \expandafter\@counttokens##1#1\@nnil shouldn't be there, should it? It will expand the first token of the argument, which isn't wanted. The code runs well without it.
@Martin: The first token may be the only one and may be a macro to be expanded. Eg, \def\test{abc de\par} \counttokens\test. If the first token isn't expandable (such as plain text or nonactive characters), then \expandafter has no effect. The extra space introduced in the definition of \counttokens was to guide against the submission of empty argument to \counttokens, eg, \counttokens{} or \counttokens\@empty, both of which correctly return 0.

Joseph Wright · Answer 2 · 2011-02-03 23:01:40Z

Depends on whether you want to count spaces. If you don't mind skipping spaces, then it is relatively easy. Using e-TeX, an expandable approach can be generated:

\catcode`\@=11\relax
\def\counttokens#1{%
  \number\numexpr\expandafter\counttokens@aux@i#1\@stop\@nil0\relax
}
\long\def\counttokens@aux@i#1{%
  \ifx#1\@stop
    \expandafter\cleanup@nil
  \else
    1 +
    \expandafter\counttokens@aux@i
  \fi
}
\long\def\cleanup@nil#1\@nil{}
\def\@stop{\@stop}
\def\test{abc de\par}
\counttokens\test
\bye

(This is essentially the expl3 function \tl_elt_count:N implemented in plain e-TeX.)

An alternative approach is not to require an expandable solution

\newcount\tokencount
\catcode`\@=11\relax
\def\:{\let\@sptoken= } \: %
\def\@stop{\@stop}
\def\counttokens#1{% 
  \begingroup
    \tokencount\z@
    \expandafter\counttoken@aux@i#1\@stop
}
\def\counttoken@aux@i{%
  \futurelet\@let@token\counttoken@aux@ii
}
\def\counttoken@aux@ii{%
  \ifx\@let@token\@stop
    \expandafter\counttoken@end
  \else
    \advance\tokencount\@ne
    \ifx\@let@token\@sptoken
      \expandafter\expandafter\expandafter\counttoken@space
    \else
      \expandafter\expandafter\expandafter\counttoken@std
    \fi
  \fi
}
\def\:{\counttoken@space} \expandafter\def\: {%
  \futurelet\@let@token\counttoken@aux@ii
}
\long\def\counttoken@std#1{%
  \counttoken@aux@i
}
\def\counttoken@end\@stop{%
  \expandafter\endgroup
  \number\tokencount
}
\def\test{abc  de\par}
\counttokens\test
\bye

I've not added any guard code for & tokens in \haligns, which would be needed for robustness in a real solution. (See Appendix D of The TeXbook, for example, to see what I mean.)

@Bruno. Do you mean in an expandable way? If so, the answer is 'not without LuaTeX', otherwise we'd have it in expl3!
@Joseph: I was thinking "non-expandable", but I guess it does not fit with your answer.
@Bruno. I've modified the answer to give some space counting. You might want to tap the minds of people on c.t.t with this one!
@Joseph: "one or more spaces count as one" because \test only contains one space, here (the two spaces are seen as one by TeX's mouth). On the other hand, with \edef\test{abc\space\space de\par}, we get 8 tokens. I still need to understand your solution, but it works!

Martin Scharrer · Answer 3 · 2011-02-03 22:55:30Z

Yes, using something like this: (updated version which counts also space-tokens)

\makeatletter
\newcount\tokencount
\def\counttoken#1{%
   \tokencount\z@ % set it to zero
   \def\x{\futurelet\@let@token\@counttoken}%
   \x#1\end@counttoken
}
% Unique endmarker
\def\end@counttoken{\@gobble{end@counttoken}}
% A macro which eats a space, the \@firstofone is required to
% avoid that the space in the parameter text itself is eaten:
\@firstofone{\def\eatspace} {}
% Recursive Macro => Loop
\def\@counttoken{%
   \ifx\@let@token\end@counttoken
      \let\x\relax
   \else
      \advance\tokencount by 1\relax
   \fi
   \ifx\@let@token\@sptoken
       % The space token will be eaten
       % Note that two spaces will still be combined by TeX!
       % If this is not what is wanted the catcode of space
       % must be changed beforehand, e.g. using \obeyspaces!
       \expandafter\expandafter
       \expandafter\x
       \expandafter\eatspace
   \else
       % Eat the next non-space token now, then call \x
       \afterassignment\x
       \expandafter\let\expandafter\@let@token\expandafter=%
   \fi
}

Test:

\counttoken{A{something}\beta}
\showthe\tokencount
% Prints:> 13
\counttoken{123}
\showthe\tokencount
% Prints:> 3

Basically the next token is \let to \@let@token and afterwards (\afterassignment) the token is counted and checked against the endmarker. Then the macro calls itself until the end marker is reached. Counts all tokens except of course the endmarker. You should add some grouping at the correct places, which depend on the exact way you apply the code.

BUT

if you really just want to count macro arguments, i.e. A{something}B would count as 3:

\newcount\argcount
\def\countargs#1{%
   \argcount\@z % set it to zero
   \@countargs#1\end@countargs
}
% Unique endmarker
%\def\end@countargs{\@gobble{end@countargs}} % doesn't really have to be defined; never executed
\def\end@@countargs{\end@countargs}
% Recursive Macro => Loop
\def\@countargs#1{%
   \def\@tempa{#1}%
   \ifx\@tempa\@end@@counttoken\else
      \advance\tokencount by 1\relax
      \expandafter\@countargs
   \fi
}

in your first code snippet, typos: \@z -> \z@ and \@end@counttoken -> \end@counttoken. Also, spaces are skipped after a \let.
@Bruno: Thanks for pointing out the typos, I will fix them. I assumed the same behavior from \let as from \futurelet which doesn't skip spaces. That's why \@ifnextchar goes to some trouble to skip them manually.
@Bruno: I know updated the first code to also count space-tokens by using \futurelet instead. See also the comments in the code.
that works indeed! (and I might be able to use it for my question on a token list's head)

Ahmed Musa · Answer 4 · 2011-02-08 17:53:46Z

Counting non-space arguments separately:

\documentclass{article}
\usepackage{catoptions}
\makeatletter
\newvariables{count}{}{argcount,spacecount,nonspacecount}[\m@ne]
\advance\nonspacecount\@ne
\def\countargs#1{%
  \def\countargs##1{%
    \begingroup
    \def\end@countargs{\end@countargs}%
    \def\countargs@a{\futurelet\next\countargs@c}%
    \def\countargs@b####1{\countargs@a}%
    \expandafter\countargs@a##1#1\end@countargs
  }%
}
\countargs{ }
\def\countargs@c{%
  \ifxTF\next\end@countargs{%
    \edef\next{%
      \endgroup\argcount\the\argcount\relax
      \spacecount\the\spacecount\relax
      \nonspacecount\the\nonspacecount\relax
    }%
    \expandafter\next\@gobble
  }{%
    \advance\argcount\@ne
    \ifxTF\next\@sptoken{%
      \advance\spacecount\@ne
      \afterassignment\countargs@a\let\next= %
    }{%
        \advance\nonspacecount\@ne\countargs@b
    }%
  }%
}

% Test
\def\reserved@a#1{{#1#1#1A#1{something}#1\beta#1}}
\expandafter\countargs\reserved@a{ }
\edef\result{%
  \string\argcount: \the\argcount,\@space
  \string\spacecount: \the\spacecount,\@space
  \string\nonspacecount: \the\nonspacecount
}
\show\result

\end{document}

Ahmed Musa · Answer 5 · 2011-02-07 21:37:17Z

Martin’s \countargs macro will not count spaces in the argument because leading spaces may be ignored when a macro grabs an argument. If we want to count spaces in the argument (I can’t see a need for that now but it is an interesting scenario), then the following code should work:

\documentclass{article}
\usepackage{catoptions}
\makeatletter
\newvariables{count}{}{argcount,spacecount}[\m@ne]
\def\countargs#1{%
  \def\countargs##1{%
    \begingroup
    \def\end@countargs{\end@countargs}%
    \def\countargs@a{\futurelet\next\countargs@c}%
    \def\countargs@b####1{\countargs@a}%
    \expandafter\countargs@a##1#1\end@countargs
  }%
}
\countargs{ }
\def\countargs@c{%
  \ifxTF\next\end@countargs{%
    \edef\next{%
      \endgroup\argcount\the\argcount\relax
      \spacecount\the\spacecount\relax
    }%
    \expandafter\next\@gobble
  }{%
    \advance\argcount\@ne
    \ifxTF\next\@sptoken{%
      \advance\spacecount\@ne
      \afterassignment\countargs@a
     \let\next= %
    }{%
      \countargs@b
    }%
  }%
}

% Test
\def\reserved@a#1{{#1#1#1A#1{something}#1\beta#1}}
\expandafter\countargs\reserved@a{ }
\edef\result{%
  \string\argcount: \the\argcount,\@space
  \string\spacecount: \the\spacecount
}
\show\result

\end{document}

asked	2 years ago
viewed	412 times
active	2 years ago

Counting tokens

5 Answers

Your Answer

Not the answer you're looking for? Browse other questions tagged macros tex-core or ask your own question.

Linked

Counting tokens

5 Answers

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged macros tex-core or ask your own question.

Linked

Related