Removing and storing the head of a token list/\toks

I am given a token list (parameter-less macro) \myTokenList, defined, e.g. by

\def\myTokenList{$(x + y)^2$, \ie, $$x^2+y^2+2xy$$}


and I want to split it as the first token, and the rest, obtaining the same result as

\def\head{$} \def\tail{(x + y)^2$, \ie, $$x^2+y^2+2xy$$}


In most cases, this is easy using the \split macro below:

\def\split#1#2\EndMark{\def\head{#1}\def\tail{#2}}
\expandafter\split\myTokenList\EndMark


But this fails when the first token is a space. In fact, it even gobbles an arbitrary number of spaces.

Using \futurelet seems to help, but

• I would much prefer \head to be a macro which expands to exactly the token (not something let to it)

• I don't know how to just remove the first token, even after \futurelet-ting

Note that I have no control on the token list that I am given (braces will be treated differently): this is in the context of a generic debugging package for TeX's stomach.

-
See my updated answer on "Counting tokens", it applies (at least partially) also here. If you want to see other code doing something similar, I can recommend the source code of my 'tikz-timing' package. I'm doing futurelet-scanning there as well and also wanted the next token as macro content and not as token. Something which I coudn't achieve, but I found a workaround for that. –  Martin Scharrer Feb 3 '11 at 23:20
@Martin: that does seem similar to what I want. I think I have a (slow) way out: \futurelet\next, check the category code using \ifcat\next\bgroup, \ifcat\next\egroup etc., grab the last character of \meaning\next (this has the right character code), and build the right explicit character <token> (character code and catcode) using a \lowercase trick. –  Bruno Le Floch Feb 4 '11 at 7:09
To distinguish between explicit and implicit characters, I can put append <token> to the list, and build a macro \mytest#1<token>{...} where ... tests whether #1 is empty or not. Then \mytest\tokenList<token> with the right amount of expansion will tell me whether the first occurence of <token> was at the start of \tokenList or not (<token> at the end just prevents runaway arguments). Does it seem realistic? –  Bruno Le Floch Feb 4 '11 at 7:11

One possible approach is to use \futurelet to tell if there is a space, then branch accordingly.

\catcode\@=11\relax
\def\:{\let\@sptoken= } \: %
\long\def\firstofone#1{#1}
\def\splitter#1{%
\let\splitter@arg#1%
\expandafter\futurelet\expandafter\@let@token\expandafter\splitter@aux#1%
}
\def\splitter@aux{%
\ifcat\noexpand\@let@token\@sptoken
\splitter@aux@ii
\else
\expandafter\splitter@aux@v
\fi
}
\def\splitter@aux@ii{%
\expandafter\splitter@aux@iii\splitter@arg\@nil
}
\def\splitter@aux@iii{%
\afterassignment\splitter@aux@iv
\firstofone{\let\@let@token= }%
}
\def\splitter@aux@iv#1\@nil{%
\def\tail{#1}%
}
\def\splitter@aux@v{%
\expandafter\splitter@aux@vi\splitter@arg\@nil
}
\long\def\splitter@aux@vi#1#2\@nil{%
\def\tail{#2}%
}
\catcodeq=10\relax
\def\test{q\space abc}
\splitter\test

@Joseph: Thank you :)! Unfortunately, \edef\test{\space\space abc} will lead to \head={ } and \tail={abc} (no space in the tail). Is it possible to avoid that? Also, this will fail if users introduce some tokens with catcode 10 but character code different from space. For that, I will need to analyse the \meaning a bit more. –  Bruno Le Floch Feb 3 '11 at 22:29
@Bruno. Multiple spaces are really awkward (see also my answer to tex.stackexchange.com/questions/10205). As I've said there, I'd bring up the challenge on c.t.t. On awkward character codes, you are probably right that \meaning might help out, possibly combined with \if to test if you are dealing with a space or something else (as you know that \meaning\splitter@arg will start with a catcode 12 token if you have some character with catcode 10 in the original and it's not a space). –  Joseph Wright Feb 3 '11 at 22:55
@Bruno: You would need to change the catcode of spaces first to something else as 10 (space) to avoid that TeX is combining them. E.g. \obeyspaces makes them active and lets them expand to a printable space :-). The same applies to line endings. –  Martin Scharrer Feb 3 '11 at 23:26
@Martin: not quite: \edef\twospaces{\space\space} expands to two space tokens with catcode 10. Since my token lists arise after multiple expansions and various steps in an arbitrary LaTeX code, anything can happen. –  Bruno Le Floch Feb 3 '11 at 23:40