I am given a token list (parameter-less macro) \myTokenList, defined, e.g. by

\def\myTokenList{$(x + y)^2$, \ie, \(x^2+y^2+2xy\)}

and I want to split it as the first token, and the rest, obtaining the same result as

\def\tail{(x + y)^2$, \ie, \(x^2+y^2+2xy\)}

In most cases, this is easy using the \split macro below:


But this fails when the first token is a space. In fact, it even gobbles an arbitrary number of spaces.

Using \futurelet seems to help, but

  • I would much prefer \head to be a macro which expands to exactly the token (not something let to it)

  • I don't know how to just remove the first token, even after \futurelet-ting

Note that I have no control on the token list that I am given (braces will be treated differently): this is in the context of a generic debugging package for TeX's stomach.

  • See my updated answer on "Counting tokens", it applies (at least partially) also here. If you want to see other code doing something similar, I can recommend the source code of my 'tikz-timing' package. I'm doing futurelet-scanning there as well and also wanted the next token as macro content and not as token. Something which I coudn't achieve, but I found a workaround for that. Feb 3, 2011 at 23:20
  • @Martin: that does seem similar to what I want. I think I have a (slow) way out: \futurelet\next, check the category code using \ifcat\next\bgroup, \ifcat\next\egroup etc., grab the last character of \meaning\next (this has the right character code), and build the right explicit character <token> (character code and catcode) using a \lowercase trick. Feb 4, 2011 at 7:09
  • To distinguish between explicit and implicit characters, I can put append <token> to the list, and build a macro \mytest#1<token>{...} where ... tests whether #1 is empty or not. Then \mytest\tokenList<token> with the right amount of expansion will tell me whether the first occurence of <token> was at the start of \tokenList or not (<token> at the end just prevents runaway arguments). Does it seem realistic? Feb 4, 2011 at 7:11

1 Answer 1


One possible approach is to use \futurelet to tell if there is a space, then branch accordingly.

\def\:{\let\@sptoken= } \: %
    \def\head{ }%
  \firstofone{\let\@let@token= }%
\def\test{q\space abc}

This needs a few refinements, for example to handle empty or single-token arguments, but the general principle should be clear.

  • @Joseph: Thank you :)! Unfortunately, \edef\test{\space\space abc} will lead to \head={ } and \tail={abc} (no space in the tail). Is it possible to avoid that? Also, this will fail if users introduce some tokens with catcode 10 but character code different from space. For that, I will need to analyse the \meaning a bit more. Feb 3, 2011 at 22:29
  • @Bruno. Multiple spaces are really awkward (see also my answer to tex.stackexchange.com/questions/10205). As I've said there, I'd bring up the challenge on c.t.t. On awkward character codes, you are probably right that \meaning might help out, possibly combined with \if to test if you are dealing with a space or something else (as you know that \meaning\splitter@arg will start with a catcode 12 token if you have some character with catcode 10 in the original and it's not a space).
    – Joseph Wright
    Feb 3, 2011 at 22:55
  • @Bruno. I'll have a crack at a better solution on the space question tomorrow.
    – Joseph Wright
    Feb 3, 2011 at 23:02
  • @Bruno: You would need to change the catcode of spaces first to something else as 10 (space) to avoid that TeX is combining them. E.g. \obeyspaces makes them active and lets them expand to a printable space :-). The same applies to line endings. Feb 3, 2011 at 23:26
  • @Martin: not quite: \edef\twospaces{\space\space} expands to two space tokens with catcode 10. Since my token lists arise after multiple expansions and various steps in an arbitrary LaTeX code, anything can happen. Feb 3, 2011 at 23:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .