Possibly overkill, but within the LaTeX3 code base Bruno Le Floch has developed the idea of a 'token list action'. This is a generic concept which can then be applied in different scenarios, for example the problem posed in Are there purely expandable variants of \MakeUppercase?.
We have not provided a public interface for the generic mechanism, as it's not clear that this is needed. So here I've recoded the approach using standard TeX coding. As LaTeX3 requires \pdfstrcmp or equivalent functionality, I'm loading the pdftexmcds package to provide \pdf@strcmp.
\RequirePackage{pdftexcmds}
\makeatletter
\long\def\tl@action#1#2#3#4#5{%
\ifnum\iffalse{\fi`}=\z@\fi
\tl@action@loop#5\q@action@mark\q@action@stop
{#4}#1#2#3%
\tl@action@result{}%
}
\long\def\tl@action@loop#1\q@action@stop{%
\tl@if@head@N@type{#1}
{\tl@action@normal}
{%
\tl@if@head@grouped{#1}
{\tl@action@group}
{\tl@action@space}%
}%
#1\q@action@stop
}
\long\def\tl@action@normal#1#2\q@action@stop#3#4{%
\ifx\q@action@mark#1%
\expandafter\tl@action@end
\fi
#4{#3}#1%
\tl@action@loop#2\q@action@stop
{#3}#4%
}
\long\def\tl@action@end#1\tl@action@result#2{%
\ifnum`{=\z@}\fi
\z@
#2%
}
\long\def\tl@action@group#1#2\q@action@stop#3#4#5{%
#5{#3}{#1}%
\tl@action@loop#2\q@action@stop
{#3}#4#5%
}
\expandafter\long\expandafter\def\expandafter
\tl@action@space\space#1\q@action@stop#2#3#4#5{%
#5{#2}%
\tl@action@loop#1\q@action@stop
{#2}#3#4#5%
}
\long\def\tl@action@output#1#2\tl@action@result#3{%
#2%
\tl@action@result{#3#1}%
}
\def\q@action@mark{\q@action@mark}
\long\def\tl@if@head@N@type#1{%
\ifnum\pdf@strcmp
{\unexpanded\expandafter{\@firstofone#1{}}}{\unexpanded{#1{}}}=\z@
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
}
\long\def\tl@if@head@grouped#1{%
\ifcat*\expandafter\@gobble\expandafter{\expandafter{\string#1?}}*%
\expandafter\@secondoftwo
\else
\expandafter\@firstoftwo
\fi
}
\long\def\tl@wrap#1{%
\unexpanded\expandafter{%
\romannumeral
\tl@action
\tl@wrap@normal
\tl@wrap@normal
\tl@wrap@space
{}
{#1}%
}%
}
\long\def\tl@wrap@normal#1#2{%
\tl@action@output{[#2]}%
}
\long\def\tl@wrap@space#1{%
\tl@action@output{[ ]}%
}
\makeatother
\documentclass{article}
\begin{document}
\makeatletter
\tl@wrap{abc de { } f}
\makeatother
\end{document}
In many ways, the idea of carefully scanning for spaces and groups is the same as Ryan's, although clearly the implementation here is a little different. (\tl@action can be used for many things, hence the empty and repeated arguments!)
On the 'why does this happen?' part of the question, there are two things to bear in mind. First, TeX will read multiple spaces as a single space. So something like
foo bar baz
will be read as
foo bar baz
unless something like \obeyspaces is in action. The latter has to be set up before reading material, and is not expandable.
Secondly, a macro of the form
\def\foo#1{<do stuff>}
will skip spaces after \foo when reading #1. Thus
\foo tokens
will skip the space after \foo, and read t as the first token. To avoid this, careful control of tokens is requires so that this does not happen.
