Removing and storing the head of a token list/\toks

Question

I am given a token list (parameter-less macro) \myTokenList, defined, e.g. by

\def\myTokenList{$(x + y)^2$, \ie, \(x^2+y^2+2xy\)}

and I want to split it as the first token, and the rest, obtaining the same result as

\def\head{$}
\def\tail{(x + y)^2$, \ie, \(x^2+y^2+2xy\)}

In most cases, this is easy using the \split macro below:

\def\split#1#2\EndMark{\def\head{#1}\def\tail{#2}}
\expandafter\split\myTokenList\EndMark

But this fails when the first token is a space. In fact, it even gobbles an arbitrary number of spaces.

Using \futurelet seems to help, but

• I would much prefer \head to be a macro which expands to exactly the token (not something let to it)

• I don't know how to just remove the first token, even after \futurelet-ting

Note that I have no control on the token list that I am given (braces will be treated differently): this is in the context of a generic debugging package for TeX's stomach.

Answer 1

One possible approach is to use \futurelet to tell if there is a space, then branch accordingly.

\catcode`\@=11\relax
\def\:{\let\@sptoken= } \: %
\long\def\firstofone#1{#1}
\def\splitter#1{%
  \let\splitter@arg#1%
  \expandafter\futurelet\expandafter\@let@token\expandafter\splitter@aux#1%
}
\def\splitter@aux{%
  \ifcat\noexpand\@let@token\@sptoken
    \def\head{ }%
    \splitter@aux@ii
  \else
    \expandafter\splitter@aux@v
  \fi
}
\def\splitter@aux@ii{%
  \expandafter\splitter@aux@iii\splitter@arg\@nil
}
\def\splitter@aux@iii{%
  \afterassignment\splitter@aux@iv
  \firstofone{\let\@let@token= }%
}
\def\splitter@aux@iv#1\@nil{%
  \def\tail{#1}%
}
\def\splitter@aux@v{%
  \expandafter\splitter@aux@vi\splitter@arg\@nil
}
\long\def\splitter@aux@vi#1#2\@nil{%
  \def\head{#1}%
  \def\tail{#2}%
}
\catcode`q=10\relax
\def\test{q\space abc}
\splitter\test
\show\head
\show\tail
\bye

This needs a few refinements, for example to handle empty or single-token arguments, but the general principle should be clear.