4

In expl3, I can translate a token list \l_greet_tl into a control sequence \greet that takes two parameters as follows:

\tl_new:N
  \l_greet_tl
\tl_set:Nn
  \l_greet_tl
  { Hello,~#1!~How~is~#2~doing? }
\cs_generate_variant:Nn
  \cs_generate_from_arg_count:NNnn
  { NNnV }
\cs_generate_from_arg_count:NNnV
  \greet
  \cs_set:Npn
  { 2 }
  \l_greet_tl

Then, I can write \greet { world } { favorite~species~(humans,~presumably) } and receive the following expansion:

Hello, world! How is your favorite species (humans, presumably) doing?

However, I would now like to do the reverse, i.e. convert the replacement text of the control sequence \greet back into a token list, so that I can append to it before I convert it back into a control sequence. I can do this manually for two parameters as follows:

\tl_set:No
  \l_greet_tl
  { \greet { #1 } { #2 } }
\tl_put_right:Nn
  \l_greet_tl
  { ~Have~a~great~#3! }
\cs_generate_from_arg_count:NNnV
  \greet
  \cs_set:Npn
  { 3 }
  \l_greet_tl

Now, I can write \greet { world } { favorite~species~(humans,~presumably) } { evolutionary~leap } and receive the following expansion:

Hello, world! How is your favorite species (humans, presumably) doing? Have a great evolutionary leap!

However, I would like to be able to do this in a way that works for any number of parameters, not just two, such as:

\tl_set_from_cs:NNn
  \l_greet_tl  % token list (output)
  \greet       % control sequence (input)
  { 2 }        % parameter count
\tl_put_right:Nn
  \l_greet_tl
  { ~Have~a~great~#3! }
\cs_generate_from_arg_count:NNnV
  \greet
  \cs_set:Npn
  { 3 }
  \l_greet_tl

Unlike the existing function \cs_replacement_spec:N, the new function \tl_set_from_cs:NNn would maintain the category codes in the replacement text.

EDIT: Further clarifications (2024-04-26)

Unlike in the answer from Max Chernoff, the answer should apply to all TeX engines supported by expl3, not just LuaTeX.

Unlike in the answers from me and @egreg, doubled #s in the replacement text should be preserved. For example, assume the following redefinition of the control sequence \greet:

\cs_new_protected:Npn
  \greet
  #1#2
  {
    Hello,~#1!
    \group_begin:
    \cs_set:Npn
      \greet
      ##1
      { ~How~is~##1~doing? }
    \greet { #2 }
    \group_end:
  }

Now, I can now write \greet { world } { favorite~species~(humans,~presumably) } and still receive the following expansion:

Hello, world! How is your favorite species (humans, presumably) doing?

However, converting the control sequence to a token list as follows no longer behaves as expected:

\tl_set:No
  \l_greet_tl
  { \greet { #1 } { #2 } }
\tl_put_right:Nn
  \l_greet_tl
  { ~Have~a~great~#3! }
\cs_generate_from_arg_count:NNnV
  \greet
  \cs_set:Npn
  { 3 }
  \l_greet_tl

This is because, \greet now contains the following replacement text:

Hello,~#1!
\group_begin:
\cs_set:Npn
  \greet
  #1
  { ~How~is~#1~doing? }
\greet { #2 }
\group_end:
~Have~a~great~#3!

As you can see, the distinction between #1 and ##1 has been lost. If I now write \greet { world } { favorite~species~(humans,~presumably) } { evolutionary~leap }, TeX will produce the following error:

Use of \greet doesn't match its definition.

Preserving the doubled #s is part of the challenge.

13
  • 2
    You can't, in general. It's not possible to access the parameter text and check the category codes therein. It should be possible if the macro only has undelimited arguments.
    – egreg
    Commented Apr 24 at 11:48
  • @egreg: Thanks for answering. The question considers the less general case, where the arguments are not delimited, and gives a working example of how the replacement text can be extracted. I am interested in the "last mile" that would make my approach independent on the number of arguments.
    – Witiko
    Commented Apr 24 at 12:11
  • @Witiko Does \tl_set_rescan:Nne \l_tmpa_tl { } { \cs_replacement_spec:N \greet } work for you? It won't work for weird catcodes, but normal cases should be mostly fine. Commented Apr 26 at 10:59
  • 1
    You cannot preserve catcodes as that information is not available from TeX (already mentioned by @egreg). See for example how etoolbox approaches patching: creating a 'reference' version of a macro based on the \meaning and current catcodes, seeing if that is \ifx equal to the one to patch, giving up if it's not.
    – Joseph Wright
    Commented Apr 26 at 11:47
  • 1
    @Witiko Probably the best way to get wider discussion is to put in a PR - other members of the team may not see this comment thread but will see a PR
    – Joseph Wright
    Commented Apr 28 at 9:48

5 Answers 5

6

For macros with undelimited arguments:

\documentclass{article}

\newcommand{\test}[2]{-#1-#2-}

\ExplSyntaxOn

\seq_new:N \l__witiko_getreplacement_seq
\tl_new:N  \l__witiko_getreplacement_tmp_tl

\seq_set_from_clist:Nn \l__witiko_getreplacement_seq
 {
  {},{#1},{},{#1}{#2},{},{#1}{#2}{#3},{},{#1}{#2}{#3}{#4},
  {},{#1}{#2}{#3}{#4}{#5},{},{#1}{#2}{#3}{#4}{#5}{#6},
  {},{#1}{#2}{#3}{#4}{#5}{#6}{#7},
  {}.{#1}{#2}{#3}{#4}{#5}{#6}{#7}{#8},
  {},{#1}{#2}{#3}{#4}{#5}{#6}{#7}{#8}{#9}
 }

\cs_new_protected:Nn \witiko_getreplacement:NN
 {
  \tl_set:Ne \l__witiko_getreplacement_tmp_tl
   {
    \seq_item:Nn \l__witiko_getreplacement_seq { \tl_count:e { \cs_parameter_spec:N #1 } }
   }
  \__witiko_getreplacement:NNV #2 #1 \l__witiko_getreplacement_tmp_tl
 }

\cs_new_protected:Nn \__witiko_getreplacement:NNn
 {
  \tl_set:No #1 { #2 #3 }
 }
\cs_generate_variant:Nn \__witiko_getreplacement:NNn { NNV }

\witiko_getreplacement:NN \test \l_tmpa_tl

\tl_show:N \l_tmpa_tl

\tl_set:Nn \l_tmpb_tl { -#1-#2- }

\tl_if_eq:NNTF \l_tmpa_tl \l_tmpb_tl { \typeout{EQUAL} } { \typeout{DIFFERENT} }

\stop

The console would print

> \l_tmpa_tl=-##1-##2-.
<recently read> }

l.36 \tl_show:N \l_tmpa_tl

?
EQUAL

I store the possible parameter texts in a sequence, which I extract the suitable item from to pass it when \tl_set:No is performed.

5
  • Thanks for the answer. I provided an extended answer based off of yours that better corresponds to the terms from the question.
    – Witiko
    Commented Apr 25 at 8:40
  • Here is an issue that I have with both our answers: If the function contains nested parameters such as ##3 and ####4, then one level of #s is removed from them. To give a concrete example, let's change line 3 from your example to \newcommand{\test}[2]{-#1-#2-##3-####4-}. Then, \tl_show:N \l_tmpa_tl at the end of your example shows \l_tmpa_tl=-##1-##2-##3-####4-., not -##1-##2-####3-########4- as one would hope. Setting a control sequence of two parameters from \l_tmpa_tl will cause TeX to issue an error message about "illegal parameter number #3". Can we do better?
    – Witiko
    Commented Apr 25 at 21:41
  • @Witiko I'm not too much interested in such plays. What would they be for?
    – egreg
    Commented Apr 25 at 21:52
  • For nested control sequence definitions. For example, I can have \def\test#1{Hello #1.\def\test##1{How is ##1 doing?}} and it's important that the body of the control sequence is preserved as-is during the round trip to a token list and back. My interest is not just academic; the implementation will be used to implement appending to token renderers in the Markdown package for TeX [1, 2]. However, nested control sequence definitions are common in token renderers, which makes this issue a blocker.
    – Witiko
    Commented Apr 26 at 6:14
  • I updated the original question with clarifications and I started a bounty for this question.
    – Witiko
    Commented Apr 26 at 11:11
5

It's possible to do this for macros with delimited arguments as well, if you're able to use LuaTeX. Getting the catcodes of a macro definition is much harder than you would expect though, so we need to use lots of dirty tricks to make this work.

\documentclass{article}

\usepackage{luacode}
\begin{luacode*}
    -----------------
    --- Constants ---
    -----------------

    local assign_toks_cmdcode = token.command_id("assign_toks")
    local car_ret_cmdcode = token.command_id("car_ret")
    local cs_token_flag = 0x1FFFFFFF
    local first_digit_chrcode = string.byte("0")
    local first_letter_chrcode = string.byte("a")
    local hash_chrcode = string.byte("#")
    local last_letter_chrcode = string.byte("z")
    local let_token = token.create("let")
    local mac_param_cmdcode = token.command_id("mac_param")
    local other_char_cmdcode = token.command_id("other_char")
    local par_end_cmdcode = token.command_id("par_end")
    local random_csname_length = 8
    local slice = table.sub
    local stop_cmdcode = token.command_id("stop")
    local tokspre_token = token.create("tokspre")


    ----------------------------
    --- Function Definitions ---
    ----------------------------

    -- Gets a table representing tokens of a macro definition
    function get_toktab_from_macro(value_csname)
        local value_token = token.create(value_csname)

        -- By default, LuaTeX only gives us the contents of a macro as a string.
        -- However, it will give us the contents of a mark node as a table of
        -- tokens, so we create a fake mark node that points to the macro's
        -- definition.
        local tmp_nd = node.direct.new("mark")
        node.direct.setprev(tmp_nd + 1, value_token.mode)
        return node.direct.getfield(tmp_nd, "mark")
    end

    -- Splits a macro definition token table into its parameters and its
    -- replacement text.
    local function split_macro_toktab(meaning_toktab)
        local stop_index
        local args_count = 0

        for i, t in ipairs(meaning_toktab) do
            -- Separator between parameters and replacement text (represented by
            -- "->" inside of \meaning).
            if t[1] == stop_cmdcode then
                stop_index = i
            -- Convert a macro parameter token in the body back into a "#"
            -- token.
            elseif t[1] == mac_param_cmdcode and t[3] == 0 then
                table.insert(
                    meaning_toktab,
                    i + 1,
                    { mac_param_cmdcode, hash_chrcode, 1 }
                )
            elseif t[1] == mac_param_cmdcode and t[3] == 1 then
                t[3] = 0
            -- Convert a macro parameter token in the body back into a <digit>
            -- token.
            elseif t[1] == car_ret_cmdcode then
                table.insert(
                    meaning_toktab,
                    i + 1,
                    { other_char_cmdcode, first_digit_chrcode + t[2], 0 }
                )
                t[2] = hash_chrcode
                t[1] = mac_param_cmdcode
            -- Convert a macro parameter token in the parameters back into a
            -- pair of tokens {"#", <digit>}.
            elseif t[1] == par_end_cmdcode then
                args_count = args_count + 1
                t[1] = mac_param_cmdcode
                t[2] = hash_chrcode
                table.insert(
                    meaning_toktab,
                    i + 1,
                    { other_char_cmdcode, first_digit_chrcode + args_count, 0 }
                )
            end
        end

        -- Split the token table
        return slice(meaning_toktab, 2,              stop_index - 1),
               slice(meaning_toktab, stop_index + 1, nil           )
    end

    -- Generates a random control sequence name.
    local function random_csname()
        local random_table = {}

        for i = 1, random_csname_length do
            local random_letter = string.char(
                math.random(first_letter_chrcode, last_letter_chrcode)
            )
            table.insert(random_table, random_letter)
        end

        return table.concat(random_table)
    end

    -- Converts a token table into a \toks token (without giving it a name).
    local function toktab_to_token(value_toktab)
        local tmp_csname = random_csname()

        -- First, we create a mark node to store the raw token in.
        local tmp_nd = node.direct.new("mark")
        node.direct.setfield(tmp_nd, "mark", value_toktab)

        -- TeX expects two levels of indirection for a \toks token, so we first
        -- point a \chardef token to the token created by the mark node.
        token.set_char(tmp_csname, node.direct.getprev(tmp_nd + 1), "global")

        -- Then, we create a \toks token that points to the \chardef token.
        return token.create(
            token.create(tmp_csname).tok - cs_token_flag,
            assign_toks_cmdcode
        )
    end

    -- \let's a token to a control sequence name.
    local function let_csname_token(name_csname, value_token)
        -- We need to create a token with the name first, otherwise we get an
        -- "undefined_cs" token which is useless.
        token.set_char(name_csname, 0)
        local name_token = token.create(name_csname)

        -- There's no way to do this directly from Lua, so we start a new TeX
        -- run and use \let to assign the token.
        tex.runtoks(function()
            token.put_next(let_token, name_token, value_token)
        end)

        return token.create(name_csname)
    end

    -- Copies a fake \toks token into a real \toks register.
    --
    -- The token created by "let_csname_token" is a semi-valid \toks register:
    -- it behaves like a \toks register with \the and similar, but it gives a
    -- (mostly harmless) error with \show and \meaning. To fix this, we copy
    -- the token's contents into a real \toks register.
    local function token_to_toksreg(toksreg_csname, value_token)
        -- Clear the register
        tex.toks[toksreg_csname] = ""

        local toksreg_token = token.create(toksreg_csname)
        local value_toksreg = let_csname_token(random_csname(), value_token)

        -- Prepend the fake \toks register onto the empty real one, giving
        -- us a real \toks register with the correct value.
        tex.runtoks(function()
            token.put_next(tokspre_token, toksreg_token, value_toksreg)
        end)
    end

    -- Registers a TeX command that calls the given Lua function.
    local function register_tex_cmd(target_csname, func, args)
        local scanners = {}
        for _, arg in ipairs(args) do
            scanners[#scanners+1] = token["scan_" .. arg]
        end

        local function scanning_func()
            local values = {}
            for _, scanner in ipairs(scanners) do
                values[#values+1] = scanner()
            end

            func(table.unpack(values))
        end

        local index = luatexbase.new_luafunction(target_csname)
        lua.get_functions_table()[index] = scanning_func
        token.set_lua(target_csname, index, "global")
    end


    --------------------
    --- TeX Commands ---
    --------------------

    register_tex_cmd("__example_macro_to_toks:N", function(value_csname)
        -- Get a table representing the macro's definition tokens
        local meaning_toktab = get_toktab_from_macro(value_csname)

        -- Split the macro into its parameters and replacement text
        local params_toktab, replacement_toktab = split_macro_toktab(meaning_toktab)

        -- Save the parameters in a \toks register
        local params_token = toktab_to_token(params_toktab)
        token_to_toksreg("l__example_parameters_toks", params_token)

        -- Save the replacement text in a \toks register
        local replacement_token = toktab_to_token(replacement_toktab)
        token_to_toksreg("l__example_replacement_toks", replacement_token)
    end, {"csname"})
\end{luacode*}

\ExplSyntaxOn
    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    %%% Variable Declarations %%%
    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%

    \exp_args_generate:n { NNNV }
    \newtoks \l__example_parameters_toks
    \newtoks \l__example_replacement_toks
    \tl_new:N \l_example_parameters_tl
    \tl_new:N \l_example_replacement_tl
    \scan_new:N \s_example


    %%%%%%%%%%%%%%%%%%%%%%%%%
    %%% Macro Definitions %%%
    %%%%%%%%%%%%%%%%%%%%%%%%%

    % Sets "\l_example_parameters_tl" with the parameters of the provided macro
    % and "\l_example_replacement_tl" with the replacement text of the same
    % macro.
    \cs_new:Nn \example_macro_to_tl:N {
        \__example_macro_to_toks:N #1
        \tl_set:NV \l_example_parameters_tl  \l__example_parameters_toks
        \tl_set:NV \l_example_replacement_tl \l__example_replacement_toks
    }

    % Defines the provided macro with parameters "\l_example_parameters_tl" and
    % replacement text "\l_example_replacement_tl".
    \cs_new:Nn \example_tl_to_macro:N {
        \exp_args:NNNNV \exp_last_unbraced:NNV \cs_set:Npn #1 \l_example_parameters_tl \l_example_replacement_tl
    }
\ExplSyntaxOff


%%%%%%%%%%%%%%%%%%%%%
%%% Demonstration %%%
%%%%%%%%%%%%%%%%%%%%%

\pagestyle{empty}

\begin{document}
    \ExplSyntaxOn
        % Define a test macro with weird parameters and body
        \cs_new:Npn \example_test:w #1 #2 \s_example {
            A B C~
            <#1>

            { #2 }
            \par $a^2 + b^\bgroup \pi^2\egroup$
        }

        % Get the parameters and replacement text of the test macro
        \example_macro_to_tl:N \example_test:w

        % Inspect the extracted token lists
        \tl_analysis_show:N \l_example_parameters_tl
        \tl_analysis_show:N \l_example_replacement_tl

        % Modify the extracted token lists
        \tl_put_right:Nn \l_example_parameters_tl { #3 }
        \tl_put_left:Nn \l_example_replacement_tl { \textbf{#3} }

        % Assemble a new macro from the modified token lists
        \example_tl_to_macro:N \test

        % Test the new macro
        \test X {\itshape y}\s_example Z

        % Compare the meanings of the original and new macros
        \par \meaning\example_test:w
        \par \meaning\test
    \ExplSyntaxOff
\end{document}

output

There are two main “tricks” to getting this to work.

  1. For this to work in general, we need the definition of the macro as a token list (with proper catcodes), not just as a Lua string. So to get the definition of a macro, we first create an empty \mark node. Next, we use some trickery to set its contents as an integer representing the macro definition. Then, when ask for the value of the \mark node, we actually get the contents of the macro, represented as a Lua table.

  2. Then to set a \toks register with the tokens from a Lua table, we use the same \mark trickery, but in reverse. Then, we create a \chardef token that points at the same token list that the \mark points at. Next, we carefully overflow the index of a \toks register so that it ends up pointing at the \chardef token. Because we carefully selected the offsets, this “fake” \toks register now acts as if it has the same value as the \mark.

We also need some code to convert the internal TeX token codes back into the form that the frontend expects and some code to move the fake \toks register into a real one, but this is relatively straigtforward.

1
  • 2
    I think my head just exploded a little.
    – Witiko
    Commented Apr 26 at 9:05
3

Here is a function \tl_set_from_cs:NNn that takes in a token list, a control sequence, and number of parameters, and assigns the replacement text of the control sequence to the token list:

\cs_new_protected:Nn
  \tl_set_from_cs:NNn
  {
    \tl_set:Nn
      \l_tmpa_tl
      { #2 }
    \int_step_inline:nn
      { #3 }
      {
        \tl_put_right:Nn
          \l_tmpa_tl
          { { #### ##1 } }
      }
    \exp_args:NNV
      \tl_set:No
      #1
      \l_tmpa_tl
  }

Unlike in the answer by egreg, no extra data structure is required. This comes at the cost of extra expansion steps, so egreg's approach may be preferable if the function is to be called frequently.

Here is a demonstration of using the function \tl_set_from_cs:NNn:

\cs_new:Npn
  \greet
  #1#2
  { Hello,~#1!~How~is~#2~doing? }
\tl_set_from_cs:NNn
  \l_greet_tl
  \greet
  { 2 }
\tl_put_right:Nn
  \l_greet_tl
  { ~Have~a~great~#3! }
\cs_generate_variant:Nn
  \cs_generate_from_arg_count:NNnn
  { NNnV }
\cs_generate_from_arg_count:NNnV
  \greet
  \cs_set:Npn
  { 3 }
  \l_greet_tl

Now, I can write \greet { world } { favorite~species~(humans,~presumably) } { evolutionary~leap } and receive the following expansion:

Hello, world! How is your favorite species (humans, presumably) doing? Have a great evolutionary leap!

As demonstrated in egreg's answer, we can also simplify the function signature to \tl_set_from_cs:NN by deducing the number of parameters from the parameter spec:

\cs_new_protected:Nn
  \tl_set_from_cs:NN
  {
    \tl_set:Ne
      \l_tmpa_tl
      { \cs_parameter_spec:N #2 }
    \int_set:Nn
      \l_tmpa_int
      { \tl_count:N \l_tmpa_tl / 2 }
    \tl_set_from_cs:NNV
      #1
      #2
      \l_tmpa_int
  }
\cs_generate_variant:Nn
  \tl_set_from_cs:NNn
  { NNV }

Here is a demonstration of using the function \tl_set_from_cs:NN:

\cs_new:Npn
  \greet
  #1#2
  { Hello,~#1!~How~is~#2~doing? }
\tl_set_from_cs:NN
  \l_greet_tl
  \greet
\tl_put_right:Nn
  \l_greet_tl
  { ~Have~a~great~#3! }
\cs_generate_from_arg_count:NNnV
  \greet
  \cs_set:Npn
  { 3 }
  \l_greet_tl

Now, I can write \greet { world } { favorite~species~(humans,~presumably) } { evolutionary~leap } and receive the following expansion again:

Hello, world! How is your favorite species (humans, presumably) doing? Have a great evolutionary leap!

If the function \tl_set_from_cs:NN were to be added to the l3tl module of expl3, other variants should also be added, such as \tl_gset_from_cs:NN.

Furthermore, it should be clearly documented that only control sequences with undelimited parameters can be converted to token lists this way.

EDIT: Further clarifications (2024-04-27)

We can preserve doubled #s in the replacement text as follows:

  1. Expand the control sequence using as parameters control sequences such as \witiko_parameter_1, \witiko_parameter_2, \witiko_parameter_3, etc. that are unlikely to appear in the replacement text.
  2. Double the #s in the result of the expansion.
  3. Replace \witiko_parameter_1, \witiko_parameter_2, \witiko_parameter_3, etc. in the result of the expansion with #1, #2, #3, etc.

Here is a concrete implementation:

\cs_new_protected:Nn
  \tl_set_from_cs:NNn
  {
    \tl_set:Nn
      \l_tmpa_tl
      { #2 }
    \int_step_inline:nn
      { #3 }
      {
        \exp_args:NNc
          \tl_put_right:Nn
          \l_tmpa_tl
          { witiko_parameter_ ##1 }
      }
    \exp_args:NNV
      \tl_set:No
      \l_tmpb_tl
      \l_tmpa_tl
    \regex_replace_all:nnN
      { \cP. }
      { \0\0 }
      \l_tmpb_tl
    \int_step_inline:nn
      { #3 }
      {
        \regex_replace_all:nnN
          { \c { witiko_parameter_ ##1 } }
          { \cP\# ##1 }
          \l_tmpb_tl
      }
    \tl_set:NV
      #1
      \l_tmpb_tl
  }

Here is a demonstration of the implementation:

\cs_new_protected:Npn
  \greet
  #1#2
  {
    Hello,~#1!
    \group_begin:
    \cs_set:Npn
      \greet
      ##1
      { ~How~is~##1~doing? }
    \greet { #2 }
    \group_end:
  }
\tl_set_from_cs:NN
  \l_greet_tl
  \greet
\tl_put_right:Nn
  \l_greet_tl
  { ~Have~a~great~#3! }
\cs_generate_from_arg_count:NNnV
  \greet
  \cs_set:Npn
  { 3 }
  \l_greet_tl

Now, I can now write \greet { world } { favorite~species~(humans,~presumably) } { evolutionary~leap } and receive the following expansion:

Hello, world! How is your favorite species (humans, presumably) doing? Have a great evolutionary leap!

This is as expected.

4
  • Both your \tl_set_from_cs:NNn and your \greet using a nested definition should get defined protected, so use \cs_new_protected:Npn instead of \cs_new:Npn.
    – Skillmon
    Commented Apr 27 at 21:56
  • @Skillmon: Why should the functions be defined as protected?
    – Witiko
    Commented Apr 27 at 22:22
  • 2
    Because they use assignments and aren't fully expandable. Everything that's not fully expandable should be defined protected.
    – Skillmon
    Commented Apr 27 at 22:49
  • @Skillmon: Thanks, I updated both the question and my answer to use \cs_new_protected:Nn and \cs_new_protected:Npn where appropriate.
    – Witiko
    Commented Apr 28 at 7:53
2

Another solution using etl. This has a few (one?) upsides and downsides compared to using l3regex, i.e.:

  1. faster (compared to Witiko's answer with precompiled regexes: roughly 17 times faster; compared to your answer with precompiled regexes: roughly 39 times faster)
  2. normalises category 1 and 2 tokens to {}
  3. the macros you work on must not contain internals of the etl package (because that might break \etl_act:nnnnn)
  4. the macros you work on must not contain a category 6 digit (because we use that as the marker for parameters)

If point 2 doesn't bother you, here's the code:

\documentclass{article}

\usepackage{etl}

\ExplSyntaxOn

% act on the argument to double category-6-#, and turn a category-6-digit into
% the appropriate parameter
\cs_new:Npn \__witiko_tl_set_from_cs_fix_params:n
  {
    \etl_act:nnnnn
      \__witiko_tl_set_from_cs_params_N:nN
      { ~ \use_none:n }
      \__witiko_tl_set_from_cs_params_n:nn
      {}
  }
% recurse into groups
\cs_new:Npn \__witiko_tl_set_from_cs_params_n:nn #1#2
  { { \__witiko_tl_set_from_cs_fix_params:n {#2} } }
% check if token is a parameter token, if so check if it's a number
\cs_new:Npn \__witiko_tl_set_from_cs_params_N:nN #1#2
  {
    \token_if_eq_catcode:NNTF ## #2
      {
        % change digits to parameters
        \str_case:nnF {#2}
          {
            { 11 } { ##1 }     { 22 } { ##2 }     { 33 } { ##3 }
            { 44 } { ##4 }     { 55 } { ##5 }     { 66 } { ##6 }
            { 77 } { ##7 }     { 88 } { ##8 }     { 99 } { ##9 }
          }
          % double every other parameter token
          { #2#2 }
      }
      { \exp_not:N #2 }
  }
% This implementation works as long as #1 is a single digit, which should always
% be the case but we don't explicitly test for it.
\cs_new:Npn \__witiko_tl_set_from_cs_param:n #1
  { \char_generate:nn { `#1 } { 6 } }
\cs_new_protected:Npn \witiko_tl_set_from_cs:NNn #1#2#3
  {
    \tl_set:Ne #1
      {
        % first build the argument list, then expand the macro once, and finally
        % turn our parameters back into the correct form
        \exp_args:NNe \exp_args:No \__witiko_tl_set_from_cs_fix_params:n
          {
            \exp_not:n { #2 }
            \int_step_function:nN {#3} \__witiko_tl_set_from_cs_param:n
          }
      }
  }

\tl_new:N \l_greet_tl
\cs_generate_variant:Nn \cs_generate_from_arg_count:NNnn { NNnV }

\cs_new:Npn
  \greet
  #1#2
  {
    Hello,~#1!
    \group_begin:
    \cs_set:Npn
      \greet
      ##1
      { ~How~is~##1~doing? }
    \greet { #2 }
    \group_end:
  }

\witiko_tl_set_from_cs:NNn \l_greet_tl \greet {2}

\tl_put_right:Nn
  \l_greet_tl
  { ~Have~a~great~#3! }
\cs_generate_from_arg_count:NNnV
  \greet
  \cs_set:Npn
  { 3 }
  \l_greet_tl

\ExplSyntaxOff

\begin{document}
\greet{world}{favorite species (humans, presumably)}{evolutionary leap}
\end{document}
3
  • How do you test the speed of this? I'm slightly curious as to how this compares to my (insane) Lua-based solution. Commented Apr 28 at 0:22
  • 1
    @MaxChernoff l3benchmark makes benchmarking really simple :) (just be sure to run on a Linux box, my experience on Windows machines is that the measurements are quite inconsistent; I have no experience on Mac)
    – Skillmon
    Commented Apr 28 at 0:33
  • @MaxChernoff I ran the benchmark, your Lua solution is about 3 times faster.
    – Skillmon
    Commented Apr 28 at 0:38
1
+300

We can make this work with doubled parameter characters (#), with delimited arguments, and with any engine, with two restrictions:

  1. The target macro cannot use "FF as its macro parameter character.

  2. Any delimiters must have “normal” catcodes.

I would expect that >99% of macros meet these requirements, so this shouldn't be too restrictive in practice.

\documentclass{article}

\pagestyle{empty}
\parindent=0pt

\ExplSyntaxOn

%%%%%%%%%%%%%%%%%%%
%%% Definitions %%%
%%%%%%%%%%%%%%%%%%%

%% Copies the replacement text of a macro (#1) into a token list (#2).
\cs_new_protected:Nn \example_cs_to_tl:NN {
    %% Get the parameters used by the macro
    \tl_set:Ne #2 { \cs_parameter_spec:N #1 }

    %% Convert the parameters into normal catcodes to handle any delimited
    %% arguments.
    \tl_set_rescan:NnV #2 {} #2

    %% Use <"FF>_6 as the macro parameter character instead of the usual <#>_6.
    %% We do this so that we can distinguish between any inner macro parameters
    %% and the new ones that we're passing in here.
    \regex_replace_all:nnN { \# ( \d ) } { { \cP\xFF \1 } } #2

    %% Expand the macro to get at its replacement text.
    \tl_set:Nf #2 {
        \exp_last_unbraced:NNNo \exp_after:wN \exp_stop_f: #1 #2
    }

    %% Double all the original parameter characters, ignoring our new ones.
    \regex_replace_all:nnN { \cP [^ \xFF ] } { \0 \0 } #2
}

%% I've used inline regexes here to make the code easier to follow, but you
%% should use `\regex_const:Nn` in the real code since it's considerably faster.

\cs_generate_variant:Nn \cs_generate_from_arg_count:NNnn { NNnV }


%%%%%%%%%%%%%%%%%%%%%
%%% Demonstration %%%
%%%%%%%%%%%%%%%%%%%%%

%% A test macro with two normal arguments and some inner doubled #'s.
\cs_new:Npn \original #1 #2 {
    Hello,~#1!~
    \group_begin:
        \cs_set:Npn \original ##1 {
            ~How~is~##1~doing?

            \cs_gset:Npn \inner ####1 {
                #1~##1~####1.
            }
        }
        \original { #2 }
    \group_end:
}

%% Extract the replacement text and define a new macro with the same body.
\example_cs_to_tl:NN \original \l_tmpa_tl
\cs_generate_from_arg_count:NNnV \new \cs_set:Npn { 2 } \l_tmpa_tl

%% A test macro with some weird delimited arguments.
\cs_new:Npn \weirddelims #1, #2 #3 ABC \relax {
    <#2>
    \def \inner ##1 #3 {
        <#1>
    }
}

%% Extract the replacement text and define a new macro with the same body.
\example_cs_to_tl:NN \weirddelims \l_tmpa_tl
\cs_generate_from_arg_count:NNnV \newweirddelims \cs_set:Npn { 3 } \l_tmpa_tl

\ExplSyntaxOff

%% Show the comparison between the original and new macros.
\begin{document}
    \texttt{\small \meaning\original}%
    \par
    \original{AAA}{BBB} \inner{CCC}
    \bigskip

    \texttt{\small \meaning\original}%
    \par
    \new{DDD}{EEE} \inner{FFF}
    \bigskip

    \texttt{\small \meaning\weirddelims}%
    \par
    \weirddelims{GGG},{HHH}{III}ABC\relax \inner{JJJ}III
    \bigskip

    \texttt{\small \meaning\newweirddelims}%
    \par
    \newweirddelims{KKK}{LLL}{MMM} \inner{NNN}MMM
\end{document}

output

The trick here is to use a different macro parameter character when expanding the target macro so that we can distinguish between the “inner” nested parameters and the outermost “real” parameters.

9
  • Can't we use \regex_replace_all:nnN { \cP. } { \0 \0 } #2 instead, so that we match and double any characters with catcode 6, not just #?
    – Witiko
    Commented Apr 27 at 18:43
  • This is a more general version of what I came up with earlier today and already added to my answer. Namely, I considered only control sequences with undelimited parameters. Furthermore, I used control sequences \witiko_parameter_1, \witiko_parameter_2, etc. instead of <"FF>_1, <"FF>_2, etc. to expand the control sequence. This felt safer as these seem less likely to appear in the replacement text by chance. Regardless, I can't award the bounty to myself, so it is yours, congrats!
    – Witiko
    Commented Apr 27 at 18:44
  • 1
    I think this depends on an implementation detail of \exp_last_two_unbraced:Noo, namely that it expands the third argument before it expands the second one and that the second one actually "sees" the result, which is not documented behaviour, for instance it could be implemented as \cs_new:Npn \exp_last_two_unbraced:Noo #1#2#3 { \use:e { \exp_not:N #1 \exp_not:o {#2} \exp_not:o {#3} }.
    – Skillmon
    Commented Apr 27 at 21:25
  • 1
    (Re: \exp_last_...) That's still incorrect though, V expands the value of a single macro and you're again using an implementation detail :) In your usage I'd argue that it should be \exp_last_unbraced:NNNo \exp_after:wN.
    – Skillmon
    Commented Apr 28 at 0:32
  • 1
    @MaxChernoff: Using \0 \0 instead of \cP\# \cP\# seems less destructive, as it maintains not just the original category codes but also the original characters in the replacement text.
    – Witiko
    Commented Apr 28 at 7:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .