1

I have a question concerning the decomposition of macro definitions:

In chapter XV, section 7 in the interface3 manual is a list of possible decompositions:

For example, there is a command \token_get_prefix_spec:N listed in the interface3 manual. As described, it leaves the prefixes in the input-stream.

A token defined by \cs_new_protected:Nn \someToken:n { do~something } would thereby leave "\protected".

The other two, \token_get_arg_spec:N and \token_get_replacement_spec:N, leave the raw arguments or the replacement. All three of them work well with a \l_peek_token as "input".

But is there a method to leave the token name? The \token_to_str:N macro is not an option, as i want to know the (once expanded) name of a \l_peek_token: \token_to_str:N \l_peek_token just returns "\l_peek_token" and not "\someToken:n" (if the peeked at token is "\someToken:n").

Here is a small example showing what I want (compile with XeTeX):

\documentclass[varwidth,border=10pt]{standalone}
\usepackage{expl3}
\usepackage{newunicodechar}
\ExplSyntaxOn

\cs_new_protected:Nn \someToken:n {
    \peek_catcode:NF ## {
        char: #1\\      
        -~Meaning~of~next~token:~\token_to_meaning:N \l_peek_token\\
        -~Prefix~of~next~token:~\token_get_prefix_spec:N \l_peek_token\\
        -~Name~of~next~token:~\token_to_str:N \l_peek_token\\ % this is not working as wanted, I want \someToken:n !
    }
}

\newunicodechar{➀}{\someToken:n{a}}
\newunicodechar{➁}{\someToken:n{b}}
\ExplSyntaxOff
\begin{document}
➁➀
…
\end{document}

(This question came up while I was searching for solutions for my other questions Compare macro names instead of meaning and Peek ahead and process characters.)

4
  • 1
    That is not possible I guess, because \l_peek_token is just \let to the next token, i.e. it is assigned the same \meaning but the original “name” is lost. Commented Oct 22, 2017 at 11:52
  • @HenriMenke But how can \token_to_meaning:N \l_peek_token return the meaning (which includes the macro name: "\protected macro:->\someToken:n {a}") if it's lost? Could I somehow parse the meaning in such a way, that everything between ">" and "{" is "returned"?
    – genericFJS
    Commented Oct 22, 2017 at 11:55
  • That's not what's happening: ➁ is made active, and searches its next token; the next token is ➀, now that's the info you loose when you use peek functions, you no longer see the “➀” but have something like \let\peektoken➀; the info that's inside ➀ is what you now have, and inside it you have \sometoken:n {a}. It's not clear to me what you want to do with that?
    – Manuel
    Commented Oct 22, 2017 at 13:22
  • I want to peek and remove the following token, if the token name of the "info inside" is \someToken:n. The use case is mentioned in the link below the source code: unicode replacement of composite-emoji (like flags or skin color variations). Other methods (also mentioned in my other questions/link below) don't work in some cases.
    – genericFJS
    Commented Oct 22, 2017 at 13:49

2 Answers 2

2

If you want to extract the first token inside the definition of a macro (this needs to be an user defined macro and not a primitive, otherwise there might be situations in which it gives an error).

\ExplSyntaxOn

\cs_new:Npn \fjs_string_first_delimit_q_nil:Nw #1 #2 \q_nil { \token_to_str:N #1 }

\cs_new_protected:Nn \fjs_sometoken:n
 {
  \peek_catcode:NF ##
   {
    char: ~ #1 \\      
    Meaning ~ of ~ next ~ token: ~ \token_to_meaning:N \l_peek_token \\
    \token_if_macro:NT \l_peek_token
     {
      First ~ token ~ inside ~ the ~ next ~ token: ~ 
     \exp_after:wN \fjs_string_first_delimit_q_nil:Nw \l_peek_token aaaaaaaaa \q_nil
     }
    \par
   }
}

\newunicodechar{➀}{\fjs_sometoken:n{a}}
\newunicodechar{➁}{\fjs_sometoken:n{b}}

\ExplSyntaxOff

This is not entirely safe (e.g., \def\foo{{}} will cause problems if the next token is \foo), but for now this works well in your cases ➀➁.

2
  • Thanks, that seems like exactly what I was looking for. I'm trying to understand your code and have a few questions: What's the purpose of the "aaaaaaaaa"? It seems I can just delete it and nothing changes… What's the purpose of the \q_nil? I've read the section in interface3.pdf but can't make sense of it in your function \fjs_string_first_delimit_q_nil:Nw. In \exp_after:wN it's a placeholder to be not expanded, isn't it?!
    – genericFJS
    Commented Oct 22, 2017 at 14:55
  • Just in case the next token needs to grab arguments so that it doesn't fail and has 9 possible arguments to grab in the first expansion. This works with delimited arguments, it takes the first token and gobbles all the rest until \q_nil.
    – Manuel
    Commented Oct 22, 2017 at 15:35
0

@HenriMenke (unintentionally?) gave me the idea to parse a substring of the \meaning to check if \l_peek_token is "\someToken:n". With some explanations from this answer I got the following working code.

\documentclass[varwidth,border=10pt]{standalone}
\usepackage{expl3}
\usepackage{newunicodechar}
\ExplSyntaxOn

\tl_new:N \l_my_tl
\seq_new:N \l_my_seq

\cs_generate_variant:Nn \regex_extract_once:nnNTF { nV }

\cs_new_protected:Nn \someToken:n {
    \peek_catcode:NF ## {
        char: #1\\      
        -~Meaning~of~next~token:~\token_to_meaning:N \l_peek_token\\
        \tl_set:Nx \l_my_tl { \token_to_meaning:N \l_peek_token }
        \regex_extract_once:nVNTF { macro:->\\someToken:n } \l_my_tl \l_my_seq { 
            -~Real~name~of~next~token:~\textbackslash someToken:n
        }{ 
            -~Real~name~of~next~token:~unknown
        }\\
    }
}

\newunicodechar{➀}{\someToken:n{a}}
\newunicodechar{➁}{\someToken:n{b}}
\begin{document}
➁➀
…
\end{document}

This solution seems like a hack. Unless someone tells me that there is no better solution, I can't mark this as an accepted answer :-(

4
  • "Real name of the next token" is ➀, \sometoken:n {a} is the meaning of that token.
    – Manuel
    Commented Oct 22, 2017 at 13:23
  • @Manuel That's the point. I do not want the meaning, because it contains the parameter {a}. When I check for the meaning, ➁➀ would match, but ➀➁ wouldn't. But I want to match every kind of following \someToken:n regardless of the complete meaning. In that sense \someToken:n is the "Real name of the next token" and ➀ is only a symbol. My goals become more apparent in the linked questions at the end of this question.
    – genericFJS
    Commented Oct 22, 2017 at 13:55
  • It's not. May be the token that will end up eventually expanding after the current one, but in the same sense \sometoken will expand to \peek_catcode and \peek_catcode wil expand to something else. So, I do understand what you mean, and tried to answer that in the answer, but the real name of the next token is ➀ :P
    – Manuel
    Commented Oct 22, 2017 at 14:37
  • You're right. But I'm just a self-tought amateur and not very fluent in the exact "vocabulary". Thanks for your answer :-)
    – genericFJS
    Commented Oct 22, 2017 at 14:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .