Take the 2-minute tour ×
TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. It's 100% free, no registration required.

This question led to a new package:
xpeek

I’ve been developing my xpeek package with lots of help from you folks on TeX.SX. It’s intended to help write commands which, like the familiar \xspace, look ahead into the input stream and choose what to do based on what follows. Although the interface needs to change, it has enough functionality to implement something like xspace, or the intelligent italic-correction from LaTeX's \textit.

Which means it has the same familiar limitation as \textit, best illustrated by example:

\textit{foof}\xspace. \textit{foof}\xspace!  % bad italic correction before "."
\def\nocorrlist{.,\xspace}
\textit{foof}\xspace. \textit{foof}\xspace!  % no italic correction before "!"

Thinking about the problem, it seems I need the ability to scan ahead in the input stream, ignoring tokens from one list while looking for tokens from another. In Expl3 terms, I’m hoping to define something like \peek_inlist_ignore_auxlist:nnTF.

As an initial step toward this end, I’m trying to simply scan ahead for the next input token not on the ignore list. For example:

\def\ignorelist{ ,.;:!}
\newcommand{\nextnonpunc}{\scanignoring{\ignorelist}`\scanfound'}
The next non-punctuation mark is \nextnonpunc.,;:! xyz

yielding

The next non-punctuation mark is ‘x’.,;:! xyz

(Actually, since I’m using Expl3, the syntax would be

\ExplSyntaxOn
\tl_const:Nn \ignorelist {~,.;:!}
\NewDocumentCommand { \nextnonpunc } { }
  {
    \peek_ignore_list:N \ignorelist
    `\l_peek_token'
  }
\ExplSyntaxOff

or something like that.)

So how can I implement \peek_ignore_list:N?

The direction I’m considering is to read ahead, consuming tokens. Each token read is added to a save-list and compared to the ignore-list. If it’s on the ignore-list, continue; otherwise put the save-list back on the input stream and stop.

Does this sound reasonable so far?

To consume tokens one-by-one, I built this function:

\cs_new_protected:Npn \peek_meaning_really_remove:NTF #1 #2 #3
  {
    \peek_meaning_remove:NTF #1
      { #2 }
      {
        \peek_meaning_remove:NT \l_peek_token
          { #3 }
      }
  }

but the question above still stands: How can I implement \peek_ignore_list:N?

share|improve this question
1  
it's either feasible or not, depending on what the rules are. Eg, if you need to be able to cope with potentially seeing brace groups you probably have to use \futurelet and test for \bgroup like 2e's \@ifnextchar but futurelet _consumes the token, leaving it's definition in the token that it defines, but this means that if (say) you see an a and want to put it back you don;t know it was an a character just a \csname that has been \let to a. that may or may not matter, depending on what the tokens that you put back are supposed to do, but in general you don't know that. –  David Carlisle Jul 19 '12 at 19:46
    
See my edit; I’m OK with putting things back and am willing to accept some restrictions on what can be ignored so long as “\xspace” is ignorable. –  J. C. Salomon Jul 19 '12 at 19:50
    
I fear to misunderstand you once again today, @David: What do you mean by "\futurelet consumes the token"? –  Hendrik Vogt Jul 19 '12 at 20:08
    
I meant if you go \futurelet\tmp\mymacro a then a is removed from the input stream. But as you are tying to politely point out, that's false. Maybe I'll go back to MathML:-) –  David Carlisle Jul 19 '12 at 20:15
1  
Nooooo! (But thanks for confirming that it doesn't consume the token.) –  Hendrik Vogt Jul 19 '12 at 20:17
add comment

1 Answer

up vote 5 down vote accepted

Here is my humble attempt. I'm quite convinced that there must be better ways but it seems to do what you need.

The following lines

\AddToIgnoreList{,.;:!\xspace}
\nextnonpunc,.;:!xyz

\nextnonpunc,.;:! xyz

\nextnonpunc,.;:!\xspace xyz

\nextnonpunc,.;:!{something}

give this output:

enter image description here

Here is the complete code:

\documentclass{article}
\usepackage{expl3,xparse,xspace}

\ExplSyntaxOn
% ----------------------------------------
% variables:
\tl_new:N \l_solomon_current_ignore_token_tl
\tl_new:N \l_solomon_ignorelist_tokens_tl
\tl_new:N \l_solomon_ignored_tokens_tl

% ----------------------------------------
% setup the list:
\cs_new:Npn \solomon_add_to_ignorelist:n #1
  { \tl_put_right:Nn \l_solomon_ignorelist_tokens_tl { #1 }}

\cs_new:Npn \solomon_remove_from_ignorelist:n #1
  {
    \tl_map_inline:nn { #1 }
      { \tl_remove_all:Nn \l_solomon_ignorelist_tokens_tl { ##1 } }
  }

\NewDocumentCommand \AddToIgnoreList { m }
  { \solomon_add_to_ignorelist:n { #1 } }

\NewDocumentCommand \RemoveFromIgnoreList { m }
  { \solomon_remove_from_ignorelist:n { #1 } }

% ----------------------------------------
% check if next token is part of the ignore-list; store it in
% \l_solomon_current_ignore_token_tl
\cs_new:Npn \solomon_check_ignorelist:NTF #1#2
  { \solomon_check_ignorelist_auxi:nTF { 0 } { #1 } { #2 } }

\cs_new:Npn \solomon_check_ignorelist_auxi:nTF #1#2#3
  {
    % get the first token from the list and compare:
    \solomon_get_head:NN
      \l_solomon_current_ignore_token_tl
      \l_solomon_ignorelist_tokens_tl
    \peek_meaning_remove:VTF \l_solomon_current_ignore_token_tl
      { #2 }
      {
        \solomon_check_ignorelist_auxii:nnn { #1 }
          { \solomon_check_ignorelist_auxi:VTF \l_tmpa_int { #2 } { #3 } }
          {
            \solomon_get_head:NN
              \l_solomon_current_ignore_token_tl
              \l_solomon_ignorelist_tokens_tl
            #3
          }
      }
  }

\cs_new:Npn \solomon_check_ignorelist_auxii:nnn #1#2#3
  {
    % test was successful; rotate the list and go on if there are more
    \int_set:Nn \l_tmpa_int { #1 }
    \int_incr:N \l_tmpa_int
    \tl_remove_all:NV
      \l_solomon_ignorelist_tokens_tl
      \l_solomon_current_ignore_token_tl
    \tl_put_right:NV \l_solomon_ignorelist_tokens_tl
      \l_solomon_current_ignore_token_tl
    \int_compare:nTF { \l_tmpa_int < \tl_length:V \l_solomon_ignorelist_tokens_tl }
      { #2 }
      { #3 }
  }
\cs_generate_variant:Nn \solomon_check_ignorelist_auxi:nTF { V }
\cs_generate_variant:Nn \peek_meaning_remove:NTF { V }
\cs_generate_variant:Nn \tl_remove_all:Nn { NV }
\cs_generate_variant:Nn \tl_put_right:Nn { NV }

\cs_new:Npn \solomon_get_head:NN #1#2
  { \exp_after:wN \solomon_get_head_aux:Nw \exp_after:wN #1#2 \q_stop }
\cs_new:Npn \solomon_get_head_aux:Nw #1#2#3 \q_stop
  { \tl_set:Nn #1 { #2 } }

% ----------------------------------------
% now do some testing:
\cs_new:Npn \solomon_nextnonpunc:
  {
    \solomon_check_ignorelist:NTF
      {
        \tl_put_right:NV \l_solomon_ignored_tokens_tl
          \l_solomon_current_ignore_token_tl
        \nextnonpunc
      }
      {
        (The~next~non-punctuation~mark~is:~`\l_peek_token')~
        \tl_use:N \l_solomon_ignored_tokens_tl
        \tl_clear:N \l_solomon_ignored_tokens_tl
      }
  }
\NewDocumentCommand \nextnonpunc {} { \solomon_nextnonpunc: }

\ExplSyntaxOff


\begin{document}
\AddToIgnoreList{,.;:!\xspace}
\nextnonpunc,.;:!xyz

\nextnonpunc,.;:! xyz

\nextnonpunc,.;:!\xspace xyz

\nextnonpunc,.;:!{something}

\end{document}
share|improve this answer
    
    
It's renamed in the new version –  Marco Daniel Jul 23 '12 at 9:40
    
That was impressive! I'm going to have to study this before I can incorporate it into xpeek, especially with the all the :w commands. –  J. C. Salomon Jul 24 '12 at 3:36
    
@J.C.Salomon I'm glad if I could help and curious to see an updated version of xpeek! I just realised I mispelled your name, sorry! –  cgnieder Jul 24 '12 at 8:10
    
\solomon_check_ignorelist:NTF should be \solomon_check_ignorelist:TF, since it takes two arguments. –  Bruno Le Floch Aug 3 '12 at 13:04
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.