Underscores in xparse arguments

Question

With the recent xparse update, I noticed that my macroes which used e{\sb^} as an argument no longer worked properly when using _ as an argument e.g. \testA_{a}. I tried with the solutions in SplitArgument on colon doesn't work under expl3 syntax but none seemed to work.

Here is a MWE illustrating the problem along with some of the tried solutions, but it is not the actual use case

\documentclass[fleqn]{article}

\usepackage{xparse}
\usepackage{amsmath}

\ExplSyntaxOn

\cs_new_protected:Npn \test_sub:n #1 { | \sb{#1} }
\cs_new_protected:Npn \test_sup:n #1 { | \sp{#1} }

\NewDocumentCommand{\testA}{ e{_^} }
{
    A
    \IfNoValueF{#1}{ \test_sub:n {#1} }
    \IfNoValueF{#2}{ \test_sup:n {#2} }
}

\NewDocumentCommand{\testB}{ e{\sb^} }
{
    A
    \IfNoValueF{#1}{ \test_sub:n {#1} }
    \IfNoValueF{#2}{ \test_sup:n {#2} }
}

\group_begin:
\char_set_lccode:nn { `? } { `: }
\char_set_catcode_math_subscript:n { `? }
\tex_lowercase:D {
    \group_end:
    \NewDocumentCommand{\testC}{ e{?^}}
}
{
    A
    \IfNoValueF{#1}{ \test_sub:n {#1} }
    \IfNoValueF{#2}{ \test_sup:n {#2} }
}

\group_begin:
\use:x{
    \group_end:
    \NewDocumentCommand\exp_not:N\testD{e{\tl_to_str:n { _ }^}}
}
{
    A
    \IfNoValueF{#1}{ \test_sub:n {#1} }
    \IfNoValueF{#2}{ \test_sup:n {#2} }
}

\ExplSyntaxOff

\begin{document}
    \begin{align*}
        \testA_{a} \quad \testA^{b} \quad \testA_{a}^{b} \quad \testA^{b}_{a} \\
        \testB_{a} \quad \testB^{b} \quad \testB_{a}^{b} \quad \testB^{b}_{a} \\
        \testC_{a} \quad \testC^{b} \quad \testC_{a}^{b} \quad \testC^{b}_{a} \\
        \testD_{a} \quad \testD^{b} \quad \testD_{a}^{b} \quad \testD^{b}_{a}
    \end{align*}
    Expected
    \begin{align*}
        A|_a \quad A|^b \quad A|_a|^b \quad A|_a|^b
    \end{align*}
\end{document}

and gives the output

Edit: I just ran the same code on overleaf with texlive 2017, 2018 and 2019 and all three gives the output

which shows that e{\sb^} have previously worked (I cannot say it is the expected behaviour of xparse).

Answer 1

You can't use _ in the argument for e under \ExplSyntaxOn, because _ becomes of category code 11.

The best would be to define such commands outside the scope of \ExplSyntaxOn, but you can also do it inside it:

\documentclass[fleqn]{article}

\usepackage{xparse}
\usepackage{amsmath}

\ExplSyntaxOn

\cs_new_protected:Npn \test_sub:n #1 { | \sb{#1} }
\cs_new_protected:Npn \test_sup:n #1 { | \sp{#1} }

\exp_args:Nne \NewDocumentCommand{\testA}{ e{\char_generate:nn {`_}{8}^} }
  {
    A
    \IfNoValueF{#1}{ \test_sub:n {#1} }
    \IfNoValueF{#2}{ \test_sup:n {#2} }
  }

\ExplSyntaxOff

\begin{document}

\begin{equation*}
  \testA_{a} \quad \testA^{b} \quad \testA_{a}^{b} \quad \testA^{b}_{a} \\
\end{equation*}
Expected
\begin{equation*}
        A|_a \quad A|^b \quad A|_a|^b \quad A|_a|^b
\end{equation*}

\end{document}

The trick is to generate the character with the correct category code, namely 8.

Answer 2

I don't think that it ever worked, at least I couldn't reproduce it with texlive 2018. I would setup the command outside the expl3 catcodes:

\documentclass[fleqn]{article}

\usepackage{xparse}
\usepackage{amsmath}


\ExplSyntaxOn

\cs_new_protected:Npn \test_sub:n #1 { | \sb{#1} }
\cs_new_protected:Npn \test_sup:n #1 { | \sp{#1} }

\ExplSyntaxOff
\NewDocumentCommand{\testA}{ e{_^} }
{
    A
    \IfNoValueF{#1}{ \csname test_sub:n\endcsname {#1} }
    \IfNoValueF{#2}{ \csname test_sup:n\endcsname {#2} }
}


\begin{document}
    \begin{align*}
        \testA_{a} \quad \testA^{b} \quad \testA_{a}^{b} \quad \testA^{b}_{a} \\
     \end{align*}
    Expected
    \begin{align*}
        A|_a \quad A|^b \quad A|_a|^b \quad A|_a|^b
    \end{align*}
\end{document}