13

Here is something I don't understand about \futurelet, in this case, into macro \xfl. I can successfully do the archetypal test of

\ifx\xfl x...

to test whether the next token is an x. When true, this would indicate that \xfl replaces to an x. And yet, if I do a

\detokenize\expandafter{\xfl}

I do not get an x as I would expect, but only an unexpandable \xfl.

Why is that? Are there other tests I can perform than \ifx on a \futureletted token? Can't I truly capture (and save) that token?

\documentclass{article}
\usepackage[T1]{fontenc}
\def\fltest{\futurelet\xfl\pdecide}
\def\pdecide{%
  \ifx\xfl x\relax[Next character is x]\else[Not x]\fi\par
  (Can't detokenize\detokenize\expandafter{\xfl})\par
}
\begin{document}
\fltest xyz
\end{document}

enter image description here

Where I would really like to end up is to be able to take the token which has been \futureletted and pass it as an argument to a \readlist. But this code hangs:

\documentclass{article}
\usepackage{listofitems}
\def\fltest{\futurelet\xfl\pdecide}
\def\pdecide{%
  \setsepchar{x}% SEARCH FOR x IN NEXT LIST
  \readlist\mylist{\xfl}\listlen\mylist[]:
}
\begin{document}
\fltest xyz
\end{document}

Note: siracusa rightly notes that \xfl doesn't replace with x, it is x. Nonetheless, there are things I can do with x that I cannot do with \xfl. For example, I can `x to get the ascii value of x, but cannot do a `\xfl to get the same thing.

|improve this question
  • Note that after a \let or \futurelet the control sequence is basically the same thing as the following token, just with a different name. That's why \ifx yields the true branch, and that's also why you get \xfl back when you use it with \expandafter, because \xfl already is fully expanded. – siracusa Feb 12 at 3:50
  • @siracusa Yes, I see that now, but my question still stands...what can I do other than \ifx to probe the quantity? – Steven B. Segletes Feb 12 at 3:53
  • This is similar to my question: tex.stackexchange.com/questions/474444/… (which is not yet answered). – David Purton Feb 12 at 5:33
  • @StevenB.Segletes \meaning\xfl might give you more data about what it has inside. – Manuel Feb 12 at 7:18
11

The \let and \futurelet primitives create implicit character tokens. Thus when does

\futurelet\foo\baz a

the token \foo is an a. It is not a macro which expands to an a.

One can see this by using \show and \meaning:

\def\baz{\edef\test{\meaning\foo}\show\foo\show\test}
\futurelet\foo\baz a

You can do anything with \foo that doesn't require an explicit character token. For example, if we do

\let\bgroup={

we can do

\hbox\bgroup <content>

but not

\def\foo\bgroup <definition>

One can use \meaning to disect the text, for example if we know it's a letter

\def\baz{%
  \edef\test{\meaning\foo}%
  \edef\test{\expandafter\bazaux\test\stop}%
  \show\test
}
\edef\bazaux{\def\noexpand\bazaux\detokenize{the letter }##1\noexpand\stop{##1}}
\bazaux
\futurelet\foo\baz a

(One could of course test first for this case.)

|improve this answer
  • Is there a place I can read up a bit more on the distinctions between what one can do with implicit vs. explicit tokens? It seems I am quite limited in what I can do with an implicit token. – Steven B. Segletes Feb 12 at 11:19
  • 1
    To be precise, I think the terminology The TeXbook uses is “implicit character” (and “implicit character token”), rather than “implicit token”. (That is, what's implicit is that it's a character. The contrast is with having that character present explicitly, rather than categorizing tokens as implicit and explicit. AFAICT.) – ShreevatsaR Feb 12 at 16:16
  • 1
    @ShreevatsaR You are correct: I've updated – Joseph Wright♦ Feb 12 at 16:32
  • Your answer here, tex.stackexchange.com/questions/449597/…, seems very related. – Steven B. Segletes Feb 12 at 19:23
  • You may have interest in my answer to my own question, where I provide some code of a colleague that converts implicit letter/character tokens into macros. – Steven B. Segletes Feb 12 at 21:35
2

My LaTeX colleague, Christian Tellechea, as I was asking him about the limitations of implicit character tokens, quickly shot back a macro (using the listofitems package) that he calls \implicittomacro#1 that takes an implicit token (limited to catcode 11 or 12) and turns it into a macro.

Thus, the MWE I posted at the end of my question becomes do-able with Christian's help. In it, the macro \fltest uses \futurelet to capture the undigested next token, and then can test it against a multiplicity of match characters at once (here, both x and y). This is cooler than a mere \ifx test, which can only compare against a single token at a time.

\documentclass{article}

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{listofitems}
\def\implicittomacro#1{%
\def\next{\expandafter\implicittomacroi\meaning#1\implicittomacroi#1}%
     \unless\ifcat a\noexpand#1%
         \unless\ifcat.\noexpand#1%
             \def\next{\errmessage{\string#1\space is not catcode 11 or 
12}}%
         \fi
     \fi
     \next
}
\expandafter\def\expandafter\implicittomacroi\expandafter
#\expandafter1\detokenize{er} #2\implicittomacroi#3{%
\def\implicittomacroii##1\implicittomacroii{\endgroup\def#3{##1}}%
     \begingroup\endlinechar-1\everyeof{\noexpand}%
\expandafter\implicittomacroii\scantokens{#2\implicittomacroii}%
}

\def\fltest{\futurelet\xfl\pdecide}
\def\pdecide{%
  \implicittomacro\xfl%
  \setsepchar{x||y}% SEARCH FOR x or y AS THE NEXT CHARACTER
  \readlist\mylist{\xfl}%
  \ifnum\listlen\mylist[]>1\relax%
    (Following undigested token is: \mylistsep[1])%
  \else
    (Following undigested token is NEITHER x nor y)%
  \fi
}
\begin{document}
\fltest xyz

\fltest yzx

\fltest zxy
\end{document}

enter image description here

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.