1

I have a question concerning the decomposition of macro definitions:

In chapter XV, section 7 in the interface3 manual is a list of possible decompositions:

For example, there is a command \token_get_prefix_spec:N listed in the interface3 manual. As described, it leaves the prefixes in the input-stream.

A token defined by \cs_new_protected:Nn \someToken:n { do~something } would thereby leave "\protected".

The other two, \token_get_arg_spec:N and \token_get_replacement_spec:N, leave the raw arguments or the replacement. All three of them work well with a \l_peek_token as "input".

But is there a method to leave the token name? The \token_to_str:N macro is not an option, as i want to know the (once expanded) name of a \l_peek_token: \token_to_str:N \l_peek_token just returns "\l_peek_token" and not "\someToken:n" (if the peeked at token is "\someToken:n").

Here is a small example showing what I want (compile with XeTeX):

\documentclass[varwidth,border=10pt]{standalone}
\usepackage{expl3}
\usepackage{newunicodechar}
\ExplSyntaxOn

\cs_new_protected:Nn \someToken:n {
    \peek_catcode:NF ## {
        char: #1\\      
        -~Meaning~of~next~token:~\token_to_meaning:N \l_peek_token\\
        -~Prefix~of~next~token:~\token_get_prefix_spec:N \l_peek_token\\
        -~Name~of~next~token:~\token_to_str:N \l_peek_token\\ % this is not working as wanted, I want \someToken:n !
    }
}

\newunicodechar{➀}{\someToken:n{a}}
\newunicodechar{➁}{\someToken:n{b}}
\ExplSyntaxOff
\begin{document}
➁➀
…
\end{document}

(This question came up while I was searching for solutions for my other questions Compare macro names instead of meaning and Peek ahead and process characters.)

4
  • 1
    That is not possible I guess, because \l_peek_token is just \let to the next token, i.e. it is assigned the same \meaning but the original “name” is lost. Oct 22, 2017 at 11:52
  • @HenriMenke But how can \token_to_meaning:N \l_peek_token return the meaning (which includes the macro name: "\protected macro:->\someToken:n {a}") if it's lost? Could I somehow parse the meaning in such a way, that everything between ">" and "{" is "returned"?
    – genericFJS
    Oct 22, 2017 at 11:55
  • That's not what's happening: ➁ is made active, and searches its next token; the next token is ➀, now that's the info you loose when you use peek functions, you no longer see the “➀” but have something like \let\peektoken➀; the info that's inside ➀ is what you now have, and inside it you have \sometoken:n {a}. It's not clear to me what you want to do with that?
    – Manuel
    Oct 22, 2017 at 13:22
  • I want to peek and remove the following token, if the token name of the "info inside" is \someToken:n. The use case is mentioned in the link below the source code: unicode replacement of composite-emoji (like flags or skin color variations). Other methods (also mentioned in my other questions/link below) don't work in some cases.
    – genericFJS
    Oct 22, 2017 at 13:49

2 Answers 2

2

If you want to extract the first token inside the definition of a macro (this needs to be an user defined macro and not a primitive, otherwise there might be situations in which it gives an error).

\ExplSyntaxOn

\cs_new:Npn \fjs_string_first_delimit_q_nil:Nw #1 #2 \q_nil { \token_to_str:N #1 }

\cs_new_protected:Nn \fjs_sometoken:n
 {
  \peek_catcode:NF ##
   {
    char: ~ #1 \\      
    Meaning ~ of ~ next ~ token: ~ \token_to_meaning:N \l_peek_token \\
    \token_if_macro:NT \l_peek_token
     {
      First ~ token ~ inside ~ the ~ next ~ token: ~ 
     \exp_after:wN \fjs_string_first_delimit_q_nil:Nw \l_peek_token aaaaaaaaa \q_nil
     }
    \par
   }
}

\newunicodechar{➀}{\fjs_sometoken:n{a}}
\newunicodechar{➁}{\fjs_sometoken:n{b}}

\ExplSyntaxOff

This is not entirely safe (e.g., \def\foo{{}} will cause problems if the next token is \foo), but for now this works well in your cases ➀➁.

2
  • Thanks, that seems like exactly what I was looking for. I'm trying to understand your code and have a few questions: What's the purpose of the "aaaaaaaaa"? It seems I can just delete it and nothing changes… What's the purpose of the \q_nil? I've read the section in interface3.pdf but can't make sense of it in your function \fjs_string_first_delimit_q_nil:Nw. In \exp_after:wN it's a placeholder to be not expanded, isn't it?!
    – genericFJS
    Oct 22, 2017 at 14:55
  • Just in case the next token needs to grab arguments so that it doesn't fail and has 9 possible arguments to grab in the first expansion. This works with delimited arguments, it takes the first token and gobbles all the rest until \q_nil.
    – Manuel
    Oct 22, 2017 at 15:35
0

@HenriMenke (unintentionally?) gave me the idea to parse a substring of the \meaning to check if \l_peek_token is "\someToken:n". With some explanations from this answer I got the following working code.

\documentclass[varwidth,border=10pt]{standalone}
\usepackage{expl3}
\usepackage{newunicodechar}
\ExplSyntaxOn

\tl_new:N \l_my_tl
\seq_new:N \l_my_seq

\cs_generate_variant:Nn \regex_extract_once:nnNTF { nV }

\cs_new_protected:Nn \someToken:n {
    \peek_catcode:NF ## {
        char: #1\\      
        -~Meaning~of~next~token:~\token_to_meaning:N \l_peek_token\\
        \tl_set:Nx \l_my_tl { \token_to_meaning:N \l_peek_token }
        \regex_extract_once:nVNTF { macro:->\\someToken:n } \l_my_tl \l_my_seq { 
            -~Real~name~of~next~token:~\textbackslash someToken:n
        }{ 
            -~Real~name~of~next~token:~unknown
        }\\
    }
}

\newunicodechar{➀}{\someToken:n{a}}
\newunicodechar{➁}{\someToken:n{b}}
\begin{document}
➁➀
…
\end{document}

This solution seems like a hack. Unless someone tells me that there is no better solution, I can't mark this as an accepted answer :-(

4
  • "Real name of the next token" is ➀, \sometoken:n {a} is the meaning of that token.
    – Manuel
    Oct 22, 2017 at 13:23
  • @Manuel That's the point. I do not want the meaning, because it contains the parameter {a}. When I check for the meaning, ➁➀ would match, but ➀➁ wouldn't. But I want to match every kind of following \someToken:n regardless of the complete meaning. In that sense \someToken:n is the "Real name of the next token" and ➀ is only a symbol. My goals become more apparent in the linked questions at the end of this question.
    – genericFJS
    Oct 22, 2017 at 13:55
  • It's not. May be the token that will end up eventually expanding after the current one, but in the same sense \sometoken will expand to \peek_catcode and \peek_catcode wil expand to something else. So, I do understand what you mean, and tried to answer that in the answer, but the real name of the next token is ➀ :P
    – Manuel
    Oct 22, 2017 at 14:37
  • You're right. But I'm just a self-tought amateur and not very fluent in the exact "vocabulary". Thanks for your answer :-)
    – genericFJS
    Oct 22, 2017 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.