# Peek ahead for next token not in token-list

This question led to a new package:
xpeek

I’ve been developing my xpeek package with lots of help from you folks on TeX.SX. It’s intended to help write commands which, like the familiar \xspace, look ahead into the input stream and choose what to do based on what follows. Although the interface needs to change, it has enough functionality to implement something like xspace, or the intelligent italic-correction from LaTeX's \textit.

Which means it has the same familiar limitation as \textit, best illustrated by example:

\textit{foof}\xspace. \textit{foof}\xspace!  % bad italic correction before "."
\def\nocorrlist{.,\xspace}
\textit{foof}\xspace. \textit{foof}\xspace!  % no italic correction before "!"


Thinking about the problem, it seems I need the ability to scan ahead in the input stream, ignoring tokens from one list while looking for tokens from another. In Expl3 terms, I’m hoping to define something like \peek_inlist_ignore_auxlist:nnTF.

As an initial step toward this end, I’m trying to simply scan ahead for the next input token not on the ignore list. For example:

\def\ignorelist{ ,.;:!}
\newcommand{\nextnonpunc}{\scanignoring{\ignorelist}\scanfound'}
The next non-punctuation mark is \nextnonpunc.,;:! xyz


yielding

The next non-punctuation mark is ‘x’.,;:! xyz

(Actually, since I’m using Expl3, the syntax would be

\ExplSyntaxOn
\tl_const:Nn \ignorelist {~,.;:!}
\NewDocumentCommand { \nextnonpunc } { }
{
\peek_ignore_list:N \ignorelist
\l_peek_token'
}
\ExplSyntaxOff


or something like that.)

So how can I implement \peek_ignore_list:N?

The direction I’m considering is to read ahead, consuming tokens. Each token read is added to a save-list and compared to the ignore-list. If it’s on the ignore-list, continue; otherwise put the save-list back on the input stream and stop.

Does this sound reasonable so far?

To consume tokens one-by-one, I built this function:

\cs_new_protected:Npn \peek_meaning_really_remove:NTF #1 #2 #3
{
\peek_meaning_remove:NTF #1
{ #2 }
{
\peek_meaning_remove:NT \l_peek_token
{ #3 }
}
}


but the question above still stands: How can I implement \peek_ignore_list:N?

-
it's either feasible or not, depending on what the rules are. Eg, if you need to be able to cope with potentially seeing brace groups you probably have to use \futurelet and test for \bgroup like 2e's \@ifnextchar but futurelet _consumes the token, leaving it's definition in the token that it defines, but this means that if (say) you see an a and want to put it back you don;t know it was an a character just a \csname that has been \let to a. that may or may not matter, depending on what the tokens that you put back are supposed to do, but in general you don't know that. –  David Carlisle Jul 19 '12 at 19:46
See my edit; I’m OK with putting things back and am willing to accept some restrictions on what can be ignored so long as “\xspace” is ignorable. –  J. C. Salomon Jul 19 '12 at 19:50
I fear to misunderstand you once again today, @David: What do you mean by "\futurelet consumes the token"? –  Hendrik Vogt Jul 19 '12 at 20:08
I meant if you go \futurelet\tmp\mymacro a then a is removed from the input stream. But as you are tying to politely point out, that's false. Maybe I'll go back to MathML:-) –  David Carlisle Jul 19 '12 at 20:15
Nooooo! (But thanks for confirming that it doesn't consume the token.) –  Hendrik Vogt Jul 19 '12 at 20:17

Here is my humble attempt. I'm quite convinced that there must be better ways but it seems to do what you need.

The following lines

\AddToIgnoreList{,.;:!\xspace}
\nextnonpunc,.;:!xyz

\nextnonpunc,.;:! xyz

\nextnonpunc,.;:!\xspace xyz

\nextnonpunc,.;:!{something}


give this output:

Here is the complete code:

\documentclass{article}
\usepackage{expl3,xparse,xspace}

\ExplSyntaxOn
% ----------------------------------------
% variables:
\tl_new:N \l_solomon_current_ignore_token_tl
\tl_new:N \l_solomon_ignorelist_tokens_tl
\tl_new:N \l_solomon_ignored_tokens_tl

% ----------------------------------------
% setup the list:
{ \tl_put_right:Nn \l_solomon_ignorelist_tokens_tl { #1 }}

\cs_new:Npn \solomon_remove_from_ignorelist:n #1
{
\tl_map_inline:nn { #1 }
{ \tl_remove_all:Nn \l_solomon_ignorelist_tokens_tl { ##1 } }
}

{ \solomon_add_to_ignorelist:n { #1 } }

\NewDocumentCommand \RemoveFromIgnoreList { m }
{ \solomon_remove_from_ignorelist:n { #1 } }

% ----------------------------------------
% check if next token is part of the ignore-list; store it in
% \l_solomon_current_ignore_token_tl
\cs_new:Npn \solomon_check_ignorelist:NTF #1#2
{ \solomon_check_ignorelist_auxi:nTF { 0 } { #1 } { #2 } }

\cs_new:Npn \solomon_check_ignorelist_auxi:nTF #1#2#3
{
% get the first token from the list and compare:
\l_solomon_current_ignore_token_tl
\l_solomon_ignorelist_tokens_tl
\peek_meaning_remove:VTF \l_solomon_current_ignore_token_tl
{ #2 }
{
\solomon_check_ignorelist_auxii:nnn { #1 }
{ \solomon_check_ignorelist_auxi:VTF \l_tmpa_int { #2 } { #3 } }
{
\l_solomon_current_ignore_token_tl
\l_solomon_ignorelist_tokens_tl
#3
}
}
}

\cs_new:Npn \solomon_check_ignorelist_auxii:nnn #1#2#3
{
% test was successful; rotate the list and go on if there are more
\int_set:Nn \l_tmpa_int { #1 }
\int_incr:N \l_tmpa_int
\tl_remove_all:NV
\l_solomon_ignorelist_tokens_tl
\l_solomon_current_ignore_token_tl
\tl_put_right:NV \l_solomon_ignorelist_tokens_tl
\l_solomon_current_ignore_token_tl
\int_compare:nTF { \l_tmpa_int < \tl_length:V \l_solomon_ignorelist_tokens_tl }
{ #2 }
{ #3 }
}
\cs_generate_variant:Nn \solomon_check_ignorelist_auxi:nTF { V }
\cs_generate_variant:Nn \peek_meaning_remove:NTF { V }
\cs_generate_variant:Nn \tl_remove_all:Nn { NV }
\cs_generate_variant:Nn \tl_put_right:Nn { NV }

{ \exp_after:wN \solomon_get_head_aux:Nw \exp_after:wN #1#2 \q_stop }
{ \tl_set:Nn #1 { #2 } }

% ----------------------------------------
% now do some testing:
\cs_new:Npn \solomon_nextnonpunc:
{
\solomon_check_ignorelist:NTF
{
\tl_put_right:NV \l_solomon_ignored_tokens_tl
\l_solomon_current_ignore_token_tl
\nextnonpunc
}
{
(The~next~non-punctuation~mark~is:~\l_peek_token')~
\tl_use:N \l_solomon_ignored_tokens_tl
\tl_clear:N \l_solomon_ignored_tokens_tl
}
}
\NewDocumentCommand \nextnonpunc {} { \solomon_nextnonpunc: }

\ExplSyntaxOff

\begin{document}
\nextnonpunc,.;:!xyz

\nextnonpunc,.;:! xyz

\nextnonpunc,.;:!\xspace xyz

\nextnonpunc,.;:!{something}

\end{document}

-
–  cgnieder Jul 23 '12 at 8:55
It's renamed in the new version –  Marco Daniel Jul 23 '12 at 9:40
That was impressive! I'm going to have to study this before I can incorporate it into xpeek, especially with the all the :w commands. –  J. C. Salomon Jul 24 '12 at 3:36
@J.C.Salomon I'm glad if I could help and curious to see an updated version of xpeek! I just realised I mispelled your name, sorry! –  cgnieder Jul 24 '12 at 8:10
\solomon_check_ignorelist:NTF should be \solomon_check_ignorelist:TF`, since it takes two arguments. –  Bruno Le Floch Aug 3 '12 at 13:04