Tell me more ×
TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. It's 100% free, no registration required.
up vote 11 down vote favorite
2

How does consume/absorb/remove a token from then token stream in TeX?

I'm using @ifnextchar and would like to remove the character it matches.

share|improve this question
1  
With expl3 there is \peek_charcode_remove:NTF – egreg Sep 17 '12 at 11:34

1 Answer

up vote 15 down vote accepted

\@ifnextchar uses \futurelet that leaves the token in place. The token can be removed by using \let. \afterassignment helps to get the control, after the token is assigned and removed. Example:

\documentclass{article}
\usepackage{ltxcmds}

\makeatletter
\newcommand{\removeifnextchar}[3]{%
  \begingroup
  \ltx@LocToksA{\endgroup#2}%
  \ltx@LocToksB{\endgroup#3}%
  \ltx@ifnextchar{#1}{%
    \def\next{\the\ltx@LocToksA}%
    \afterassignment\next
    \let\scratch= %
  }{%
    \the\ltx@LocToksB
  }%
}
\makeatother

\newcommand*{\removeA}{%
  \removeifnextchar{A}{[A is removed]}{[no A]}%
}

\begin{document}

\removeA ABC

\removeA abc

\end{document}

Result

\ltx@ifnextchar or \@ifnextchar skips spaces. If you want to detect spaces, then use \ltx@ifnextchar@nospace.

Removing/reinserting

In general this is not possible, because \futurelet does not differentiate between implicit and explicit tokens. For example, the next token can be an opening curly brace { or it can be the command \bgroup or a command with the meaning of \bgroup. This cannot be distinguished by \futurelet or \let. The curly brace can be removed by the \let trick above. However reinserting is much more difficult. The latter cases \bgroup or other command with the same meaning are much easier, they can be catched as macro and reinserted without need to modify the token.

The following example defines a macro \embrace that analyzes the following tokens and does different actions depending on the type of the token:

  • Letters (catcode 11) are put in parentheses.
  • Other characters (catcode 12) are put in square brackets.
  • The space is replaced by the underscore.
  • Any other token is removed.
  • \par terminates the analysis.

The example file:

\documentclass{article}
\usepackage{ltxcmds}
\usepackage{xstring}

\makeatletter

\newcommand{\embrace}{%
  \futurelet\embrace@token\@embrace
}
\newcommand{\embrace@test}[1]{%
  \ifx\embrace@token#1%
    \expandafter\ltx@firstoftwo
  \else
    \expandafter\ltx@secondoftwo
  \fi
}
\newcommand*{\@embrace}{%
  \embrace@test\@sptoken{%
    \textunderscore
    \afterassignment\embrace
    \let\embrace@token= %
  }{%
    \expandafter\IfBeginWith\expandafter*%
    \expandafter{\meaning\embrace@token}{the letter }{%
      \embrace@letter
    }{%
      \expandafter\IfBeginWith\expandafter*%
      \expandafter{\meaning\embrace@token}{the character }{%
        \embrace@character
      }{%
        \ifx\embrace@token\par
        \else
          \@latex@warning{embrace: Token [\meaning\embrace@token] removed}%
          \afterassignment\embrace
        \fi
        \let\embrace@token= %
      }%
    }%
  }%
}
\newcommand*{\embrace@letter}[1]{%
  (#1)%
  \embrace
}
\newcommand*{\embrace@character}[1]{%
  [#1]%
  \embrace
}

\begin{document}

\embrace ABC 123\par

\embrace a 1{!}#&$x% 

\end{document}

Result of \embrace

And the removed tokens:

LaTeX Warning: embrace: Token [begin-group character {] removed on input line 54.
LaTeX Warning: embrace: Token [end-group character }] removed on input line 54.
LaTeX Warning: embrace: Token [macro parameter character #] removed on input line 54.
LaTeX Warning: embrace: Token [alignment tab character &] removed on input line 54.
LaTeX Warning: embrace: Token [math shift character $] removed on input line 54.
share|improve this answer
Thanks. Could you also include the opposite, insert a single char using the current catcodes(But redefine them locally to get things to work) for completeness if it is not too difficult? – AbstractDissonance Sep 17 '12 at 3:34
@AbstractDissonance What you ask for is underspecified. You can insert a control sequence (for instance \...) with \expandafter\next\csname ...\endcsname, where \next is the next thing you want to do. For character tokens, you can use \lowercase: for instance, to get a question mark with catcode 3, you would do \begingroup\lccode`$=`?\lowercase{\endgroup\next$}. – Bruno Le Floch Sep 17 '12 at 8:12
@BrunoLeFloch I'm not sure what you mean. Suppose I remove A using Heiko's code and I want to insert it right back(as if I never removed it)... will your code do that? Obviously, I would just use ifnextchar for the same behavior but that isn't the point. The point about changing cat codes was simply so one could easily insert a special token like \!inserttoken{\} would insert a ` into the stream without throwing an error BUT ` would have the catcode that was used before the macro call. – AbstractDissonance Sep 17 '12 at 9:01
@AbstractDissonance I don't understand what you are trying to acheive: a practical application would be welcome. Where do you want to insert the token you build? Also, there is no such thing as a "backslash token (with usual catcodes)", more precisely, catcode 0 is a valid category code for a character before it is read by TeX, but TeX never creates catcode 0 tokens. – Bruno Le Floch Sep 17 '12 at 13:56
@BrunoLeFloch Well, How bout you post some code for \InsertToken so I can do \InsertToken{\}mymacro which would be equivalent to \mymacro? Or even \mym\InsertToken{a}cro – AbstractDissonance Sep 17 '12 at 21:44
show 10 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.