4

I would like to define an expl3 command that—

  1. takes, as an n-type argument, code that contains parameter numbers such as #1;
  2. has variants that take this same code as an x-type or e-type argument; and
  3. does not need to be expandable.

Here is a fairly minimal working example to illustrate such a command:

\documentclass{article}
\usepackage{expl3}
\ExplSyntaxOn
\cs_new:Nn \example:n {
    \tl_map_inline:nn { 1 2 3 } {#1}
}
\cs_generate_variant:Nn \example:n { x, e }
\begin{document}
\example:n { #1 . }
\example:e { #1 . }
% In practice, I would not actually use the `expl3'-style command in the
% document text, nor would I use the `x'-type or `e'-type variant so
% unnecessarily.
\end{document}

Each of the above invocations of \example:(n|e) typesets 1.2.3., as expected. However, if \example:e is changed to \example:x, the following error message results:

This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019/NixOS.org) (preloaded format=pdflatex 2020.10.14)  14 OCT 2020 23:06
[...]
! Illegal parameter number in definition of \l__exp_internal_tl.
<to be read again> 
                   1
l.10 \example:x { #1 . }
                        
You meant to type ## instead of #, right?
Or maybe a } was forgotten somewhere earlier, and things
are all screwed up? I'm going to assume that you meant ##.

If \example:x { #1 . } is changed to \example:x { ##1 . }, it behaves exactly as \example:e { #1 . }. This behavior is unlike that of \cs_new:Nn, \cs_new:Npn, etc., in the final argument of which parameter numbers are equally acceptable regardless of whether the argument is taken as n-type or x-type.

Now, though I am not familiar with the internals of LaTeX3 (nor especially familiar with low-level TeX), I can guess that this error occurs with a variant command generated with \cs_generate_variant:Nn but not with \cs_new:Nx because \cs_new:Nx is defined manually such that \cs_new:Nx \foo:n { #1 . } translates to something involving—

\edef\foo:n#1{#1.}

—which is fine, whereas a generated variant command taking an x-type argument is such that \example:x { #1 . } apparently (from the error message) translates to something involving—

\edef\l__exp_internal_tl{#1.}

—in which the parameter number #1 clearly does need to be escaped.

Guessing aside, as far as I see, looking over parts I–IV, interface3.pdf, in its version dated 2020-10-05, does not document any difference between \cs_new:Nx (and family) and generated variant commands in their behavior regarding accepting parameter numbers in x-type versus n-type arguments. Is this difference in behavior undocumented, or am I missing something?

My actual application is that—

  1. I have an expl3 command that wraps xparse's \NewDocumentCommand and \NewExpandableDocumentCommand to define an xparse-style command that automatically gets replaced with an expandable equivalent when hyperref needs one in its \pdfstringdef and
  2. I found it would be useful for this command-defining command to be able to take the body code of the command to be defined as an x-type or e-type as well as an n-type argument, in the manner of \cs_new:N(n|x).

(Edit to try to be maximally clear: I had hoped to define a command that would work much like \cs_new:Npn, in that it (1) would define a new command ‘D’ and (2) would be able to take the last argument as x-type rather than n-type and expand it without needing references to the parameters of ‘D’ (#1 etc.) that appear in that argument to be escaped (##1 etc.), just as \cs_new:Npx doesn't need them to be escaped.)

What I have works fine if I use an e-type rather than x-type argument, and, in practice, I suppose I'll use an e-type argument and accept inflicting the “often more than 200 times slower” performance on Debian stable.1 Given that this code is, so far, purely for my own use and I am not on Debian stable but rather, as can be seen above, on a ‘less-stable’ Linux distribution that, indeed, should have TeX Live 2020 working its way through the build farm as I write, this is more a theoretical question, but, nevertheless, what is a good way, if any exists, to work around this discrepancy in behavior without relying on any undocumented behavior, such that I would have an expl3 command as described at the beginning of this post, having a variant taking an x-type argument that allows parameter numbers to be included in the same way as in the base command's corresponding n-type argument?


1 Edit to explain this reference: Debian is a popular Linux distribution known for its ‘conservative’ software updates; its ‘stable’ version provides an example of a software distribution that (as far as I see) does not have pdfTeX 1.40.20 or newer.

Edit to add: I think this is my first time asking a question on Stack Exchange, so I apologize in advance if I've done anything wrong!

1 Answer 1

4

That is expected, yes. x is \edef under the hood, and \edef as a definition command expects that a # is either followed by a valid parameter number, or by another #. And, as you observed yourself:

If \example:x { #1 . } is changed to \example:x { ##1 . }, it behaves exactly as \example:e { #1 . }.

In “The LaTeX3 Programming Language” (expl3.pdf) it says (my emphsis):

x — Fully-expanded token or braced token list. This means that the argument is expanded as in the replacement text of an \edef, and the expansion is passed to the function as a braced token list. Expansion takes place until only unexpandable tokens are left. x-type arguments cannot be nested.
e — Fully-expanded token or braced token list which does not require doubled # tokens. This expansions is very similar to x-type but may be nested and does not require that # tokens are doubled.

Which means that the x-type expansion is roughly

\edef \l__exp_internal_tl { <tokens-to-expand> }
\l__exp_internal_tl

and that requires parameter tokens (#) to be doubled, because of the requirement of \edef on what follows a parameter token. In fact, if you try

\edef \l__exp_internal_tl { a ## b #1 c }
\show \l__exp_internal_tl

you will get precisely the error you see:

! Illegal parameter number in definition of \l__exp_internal_tl.
<to be read again> 
                   1
l.3 \edef \l__exp_internal_tl { a ## b #1
                                          c }
?

because the # is followed by 1, which is not a valid parameter number in this definition (which has no parameters before the {). Then TeX tries to fix your definition by adding the second missing #, and the output of \show is:

> \l__exp_internal_tl=macro:
-> a ## b ##1 c .

Unfortunately that is the underlying behaviour of TeX, so answering your second question: no, there is nothing you can do to avoid doubling parameter tokens in an x-type expansion. You'll have to stick to e-type or the slower emulation provided in older versions of expl3. You can check if you have the faster e-type by doing \show\expanded; if the output is something like \expanded=\expanded then you're good to go.


The difference between (I'm using the variant with p here to be explicit, but it would work the same with \cs_new:Nx)

\cs_new:Npx \foo:n #1 {<stuff>#1}
\foo:n {<arg>}

and

\cs_new:Npn \foo:n #1 {<stuff>#1}
\cs_generate_variant:Nn \foo:n { x }
\foo:x {<arg>}

is subtle, but it's of most importance.

In the first case, the <stuff>#1 (called <replacement text> in proper TeX terminology) is expanded at definition-time: the x in \cs_new:Npx expands <stuff>#1, and then defines \foo:n with that code expanded. Then when you use \foo:n, it grabs the <arg> without expansion. So far so good.

In the second case, <stuff>#1 is not expanded at definition-time. Instead, only when you execute \foo:x, it will expand <arg> instead, and as an x-type expansion it expects all # to be followed by a valid parameter number or by another #.

The commands actually indicate that to you. Notice that in the first case the x is in \cs_new:Npx, so that will trigger the expansion, and in the second case the x is in \foo:x, which will trigger expansion when \foo:x is used.


Here's a sample document that demonstrates that:

\documentclass{article}
\begin{document}

\def\txt{text}
\ttfamily
\ExplSyntaxOn

% expands V here
\cs_set:Npx \foo:n #1 { [#1] [\detokenize{#1}] }
1:\foo:n {\txt}

\par

\cs_set:Npn \foo:n #1 { [#1] [\detokenize{#1}] }
\cs_generate_variant:Nn \foo:n { x }
2:\foo:x {\txt}
%    ^ expands here

\par

3:\foo:n {\txt}
%    ^ doesn't expand
\end{document}

and the output is:

enter image description here

Here's what happens in each case:

  1. \cs_set:Npx expands at definition time, expanding \detokenize{#1} which becomes ##1, and the definition is roughly (except for catcodes) the same as \cs_new:Npn\foo:n#1{[#1][##1]}. Using \foo:n {\txt} replaces only in the first bracket, as the second one is just a string of three characters (with no special meaning).

  2. Here the definition doesn't expand, and the \detokenize remains in \foo:n. Then when you use the \foo:x variant, it expands \txt, and then passes the expansion (text) to \foo:n, which now does [text][\detokenize{text}], which just prints [text] twice as you see.

  3. Here we use the :n variant of the definition above, without pre-expanding \txt, which results in [\txt][\detokenize{\txt}]. In the first bracket, \txt is expanded as usual when TeX typesets it, but in the second, \detokenize kicks in and turns the macro \txt in the characters \txt

4
  • Thank you for responding, but I don't think you've addressed either of my actual questions (the two boldfaced passages in my post). (Also, I suspect you may have misread my "does need to be escaped" as "doesn't need to be escaped"?)
    – user570286
    Oct 15, 2020 at 6:52
  • @user570286 I edited my answer to try to make it clearer. The documentation does require some knowledge of the underlying TeX language to completely understand (there is much room for improvement). Also in your recent edit you say “just as \cs_new:Npx doesn't need them to be escaped”: it does actually. Try \cs_new:Npx \foo:n #1 { #2 # }: you'll get two errors due to # not being followed by allowed tokens. (Indeed, I misread. I fixed in the edit. Sorry about that!) Oct 15, 2020 at 10:20
  • Ah, I should have checked expl3.pdf, then. I've edited my recent edit to try to be more precise.
    – user570286
    Oct 17, 2020 at 6:08
  • @user570286 I'm not sure I fully understand your second edit. Generally, yes, you must always double parameter tokens for nested definitions (that's how TeX knows what definition they belong to): \def\a#1{\def\b##1{\def\c####1{(a=#1,b=##1,c=####1)}}} \a{x}\b{y}\c{z}, and for that very reason you have to double # on an x expansion (it is a nested definition). That said, if you manually define the x variant in terms of \cs_new:Npx (rather than Npn + pre-expansion) then it won't need additional # doubling. Oct 17, 2020 at 10:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .