Building regex from another one

Question

I'm trying to define a command where the user can define a regex, which should then be used to create a larger regex that in turn will be matched on a token list.

The naive approach to insert it from a token list obviously fails because of a wrong catcode setup:

\documentclass{article}
\usepackage{expl3}

\begin{document}
\ExplSyntaxOn

\tl_set:Nn \l_foo_tl { [a-z]+ }
\regex_const:Nn \l_foo_regex { (\w+)( \[ \u{l_foo_tl} \] ) }
\regex_show:N \l_foo_regex

\seq_new:N \l_foo_seq
\regex_extract_all:NnN \l_foo_regex { a[x], b[yy], c[zzz] } \l_foo_seq
\seq_show:N \l_foo_seq

\ExplSyntaxOff
\end{document}

outputs

+-branch
  ,-group begin
  | Match, repeated 1 or more times, greedy
  |   range [97,122]
  |   range [65,90]
  |   range [48,57]
  |   char code 95
  `-group end
  ,-group begin
  | char code 91
  | char 91, catcode 12
  | char 97, catcode 11
  | char 45, catcode 12
  | char 122, catcode 11
  | char 93, catcode 12
  | char 43, catcode 12
  | char code 93
  `-group end.

What I'd want is either something like

\regex_const:Nn \l_sub_regex { [a-z]+ }
\regex_const:Nn \l_foo_regex { (\w+)( \[ ... \] ) }

where ... somehow inserts the regex represented by \l_sub_regex (both \c{l_sub_regex} and \u{l_sub_regex} give wrong results here); or a way to convert a compiled regex back to its string representation, something like \regex_to_str:N.

Perhaps there's a way to insert it back from a token list using some \detokenize or \scantokens hackery, but I'm wondering if l3regex already provides a proper solution for this.

EDIT: I found a note in the l3regex documentation about features that are "likely to be implemented at some point in the future":

Provide a syntax such as \ur{l_my_regex} to use an already-compiled regex in a more complicated regex. This makes regexes more easily composable.

So it seems such a feature doesn't currently exist but is planned for the future.

---

(By the way, it would be really helpful if the \regex_show: functions would also print the actual ASCII representation of a character if it is in the set of printable characters. Several lines of char code XXX are harder to debug than necessary.)

---

EDIT 2: Unfortunately the accepted answer doesn't seems to work correctly when a complete regex is inserted in a capturing group. The following code reproduces the problem:

\regex_set:Nn \l_tmpa_regex { a|b|c }
\cs_show:N \l_tmpa_regex

\regex_set:Nn \l_tmpa_regex { (\y{l_tmpa_regex}) }
\regex_set:Nn \l_tmpb_regex { (a|b|c) }
\regex_show:N \l_tmpa_regex
\regex_show:N \l_tmpb_regex
\cs_show:N \l_tmpa_regex
\cs_show:N \l_tmpb_regex

\regex_extract_once:NnNTF \l_tmpa_regex {a} \l_tmpa_seq {} {}
\seq_show:N \l_tmpa_seq
\regex_extract_once:NnNTF \l_tmpb_regex {a} \l_tmpa_seq {} {}
\seq_show:N \l_tmpa_seq

Both regexes are supposed to match on the token list a but only the second one does.

The final regexes in \l_tmpa_regex and \l_tmpb_regex should be the same, as the two identical \regex_show:N outputs suggest:

> Compiled regex variable \l_tmpa_regex:
+-branch
  ,-group begin
  | char code 97 (a)
  +-branch
  | char code 98 (b)
  +-branch
  | char code 99 (c)
  `-group end.

> Compiled regex variable \l_tmpb_regex:
+-branch
  ,-group begin
  | char code 97 (a)
  +-branch
  | char code 98 (b)
  +-branch
  | char code 99 (c)
  `-group end.

But the raw internal structure reveals that there is a difference (code shortened a bit):

> \l_tmpa_regex=macro:->
\__regex_branch:n {
    \__regex_group:nnnN {
        \__regex_branch:n {
            <char a>
            \__regex_branch:n { <char b> }
            \__regex_branch:n { <char c> }
        }
    }{1}{0}\c_false_bool
}

> \l_tmpb_regex=macro:->
\__regex_branch:n {
    \__regex_group:nnnN {
        \__regex_branch:n { <char a> }
        \__regex_branch:n { <char b> }
        \__regex_branch:n { <char c> }
    }{1}{0}\c_false_bool
}

If we look at the originally inserted regex, the internal structure seems fine at the start:

> \l_tmpa_regex=macro:->
\__regex_branch:n { <char a> }
\__regex_branch:n { <char b> }
\__regex_branch:n { <char c> }

Especially because the number of \__regex_branch:n commands is the same in both result regexes, I don't think this is a limitation of the internal regex structure but of the way the accepted answer inserts one regex into another. It's probably related to brace tokens going to the wrong place.

The problem seems to be more compilcated than just shifting some braces around. When l3regex reads the definition of a branch, it has internally already pushed the sequence \__regex_branch:n { \if_false: } \fi: on the result token list. A proper fix would therefore have to check in what context it occurs (beginning/middle/end of a branch) and to modify the internal token list such that the regex to be inserted fits in correctly.

Answer 1

After a crash course on l3regex I managed to pull off two (rather stupid) ways to do what you ask. I'll post them as separate answers because they are completely different both at implementation and usage levels.

Disclaimer: This answer uses internal code of l3regex which definitely should not be used in a document so it might break as soon as something changes in the code of l3regex, so please do not use this code unless you really know what you are doing.

Inserting a token list to be compiled as a regex:

Whereas this method allows for greater flexibility, it is roughly the same as predefining a token list and pre-expanding it using usual methods (assuming you defined \regex_const:Nx with \cs_generate_variant:Nn \regex_const:Nn { Nx }):

\tl_const:Nn \c_foo_tl { a-z }
\regex_const:Nx \c_bar_regex { [\tl_use:c{c_foo_tl}]+ }

so there isn't a real benefit from implementing this, so the other approach was implemented (see Bruno's answer).

This one steps in in the compilation of a regex right after the regular expression is tokenized but before the actual compilation takes place, replacing all \y{<tl var>} by the contents of \<tl var>. It's a really nasty brute-forcing of a token list inside a regular expression, but it apparently works fine.

With this method you can insert any token list to be compiled as a regular expression using \y{<tl var>}:

\tl_const:Nn \c_foo_tl { a-z }
\tl_const:Nn \c_bar_tl { [\y{c_foo_tl}]+ }
\regex_const:Nn \c_foo_regex { (\w+)( \[ \y{c_bar_tl} \] ) }

In the example above the token list \c_bar_tl is inserted into the regular expression. \c_bar_tl itself contains a recursive call to \y{c_foo_tl}. I offer two different macros which which you can choose if you want to allow recursion or not. If you don't allow recursion, the code above matches the literal string y{c_foo_tl}. If you allow recursion, then \y{c_foo_tl} expands and inserts a-z into the expression, which is then properly compiled. Mind you that with recursion enabled, infinite recursion is also enabled, if your code does so.

I used \y instead of \ur in this one just to keep the same character as in the other answer, but it could easily be \ur or some other string.

I injnected a macro in \__regex_escape_use:nnnn which, right before the regular expression is compiled, replaces all \y{<tl var>} by the contents of \<tl var>. This requires adding a line to \__regex_escape_use:nnnn, from which the code starts. The function then proceeds doing the proper replacements (recursively or not) until it finds the end of the expression. When it does the function returns the control to the regular expression engine to do its job normally.

Because this does the expansion of the escape sequence prior to the compilation of the regular expression there are no bounds on the contents of each \<tl var> used as long as the final expression, after the expansion of all \y results in a valid regular expression. This allows you to define one token list c_foo_tl with [a- an another with \y{c_foo_tl}z]+ and it would expand to the expected [a-z]+.

The function which controls the behaviour after the expansion of an escape sequence \y is \__regex_prescan_yank_eval_continue:n. Below there are two different versions to choose from, recursive and non-recursive, as discussed earlier.

Here's the code:

\documentclass{article}
\usepackage{expl3}
\ExplSyntaxOn
\cs_gset_protected:Npn \__regex_escape_use:nnnn #1#2#3#4
  {
    \group_begin:
      \tl_clear:N \l__regex_internal_a_tl
      \cs_set:Npn \__regex_escape_unescaped:N ##1 { #1 }
      \cs_set:Npn \__regex_escape_escaped:N ##1 { #2 }
      \cs_set:Npn \__regex_escape_raw:N ##1 { #3 }
      \__regex_standard_escapechar:
      \tl_gset:Nx \g__regex_internal_tl
        { \__kernel_str_to_other_fast:n {#4} }
      \__regex_prescan_yank:N \g__regex_internal_tl % <-- Added
      \tl_put_right:Nx \l__regex_internal_a_tl
        {
          \exp_after:wN \__regex_escape_loop:N \g__regex_internal_tl
          { break } \prg_break_point:
        }
      \exp_after:wN
    \group_end:
    \l__regex_internal_a_tl
  }
\cs_new_protected:Npn \__regex_prescan_yank:N #1
  { \tl_set:Nx #1 { \exp_args:NV \__regex_prescan_yank:n #1 } }
\cs_set:Npn \__regex_tmp:w #1#2
  {
    \cs_new:Npn \__regex_prescan_yank:n ##1
      { \__regex_prescan_yank:w ##1 #1 \q_nil #2 \q_stop }
    \cs_new:Npn \__regex_prescan_yank:w ##1 #1 ##2 #2
      {
        ##1
        \quark_if_nil:nT {##2}
          { \use_none_delimit_by_q_stop:w }
        \__regex_prescan_yank_eval_continue:n {##2}
      }
  }
\exp_args:Nxx \__regex_tmp:w
  { \__kernel_str_to_other_fast:n { \y } \c_left_brace_str }
  { \c_right_brace_str }
% \cs_new:Npn \__regex_prescan_yank_eval_continue:n #1
%   { % Non-recursive
%     \exp_args:Nv \__kernel_str_to_other_fast:n { #1 }
%     \__regex_prescan_yank:w
%   }
\cs_new:Npn \__regex_prescan_yank_eval_continue:n #1
  { % Recursive
    \exp_last_unbraced:Nf \__regex_prescan_yank:w
    \exp_args:Nv \__kernel_str_to_other_fast:n { #1 }
  }
\ExplSyntaxOff
\begin{document}
\ExplSyntaxOn
\tl_const:Nn \c_foo_tl { a-z }
\tl_const:Nn \c_bar_tl { [\y{c_foo_tl}]+ }
\regex_const:Nn \c_foo_regex { (\w+)( \[ \y{c_bar_tl} \] ) }
% \regex_const:Nn \c_foo_regex { (\w+)( \[ [a-z]+ \] ) }
\regex_show:N \c_foo_regex

\seq_new:N \l_foo_seq
\regex_extract_all:NnN \c_foo_regex { a[x], b[yy], c[zzz] } \l_foo_seq
\seq_show:N \l_foo_seq
\ExplSyntaxOff
\end{document}

Answer 2

After a crash course on l3regex I managed to pull off two (rather stupid) ways to do what you ask. I'll post them as separate answers because they are completely different both at implementation and usage levels.

Disclaimer: This answer uses internal code of l3regex which definitely should not be used in a document so it might break as soon as something changes in the code of l3regex, so please do not use this code unless you really know what you are doing.

Inserting a compiled regex into another regex:

A working version of this code is now implemented in l3regex as the \ur escape sequence, and the note that once was in the documentation is no longer there:

Provide a syntax such as \ur{l_my_regex} to use an already-compiled regex in a more complicated regex. This makes regexes more easily composable.

You first need to compile a regex using \regex_const:Nn or \regex_set:Nn or something like that, and then use the compiled regex inside the new regular expression:

\regex_const:Nn \c_bar_regex { [a-z]+ } % Compile \c_bar_regex
\regex_const:Nn \c_foo_regex { (\w+)( \[ \ur{c_bar_regex} \] ) } % Insert it with \ur{...}

The version I had implemented (in the edit history of this post for those who care) had several limitations; for example it didn't allow quantifiers, which the current implementation in l3regex does:

\regex_const:Nn \c_bar_regex { [a-z] } % Compile \c_bar_regex
\regex_const:Nn \c_foo_regex { \ur{c_bar_regex}+ ) } % Insert it with \ur{...}

---

Bonus, for thee who disliketh the output of \regex_show:N

\regex_show:(N|n) was also changed and now if the character code is within the visible ASCII range, the char code is printed followed by the actual character, in parentheses, so now \regex_show:n { (\w+)( \[ [a-z]+ \] ) } prints:

+-branch
  ,-group begin
  | Match, repeated 1 or more times, greedy
  |   range [97 (a),122 (z)]
  |   range [65 (A),90 (Z)]
  |   range [48 (0),57 (9)]
  |   char code 95 (_)
  `-group end
  ,-group begin
  | char code 91 ([)
  | range [97 (a),122 (z)], repeated 1 or more times, greedy
  | char code 93 (])
  `-group end.

Answer 3

This is now implemented (in the upcoming release) as \ur{l_my_regex}:

\regex_set:Nn \l_tmpa_regex { a|b|c* }
\regex_set:Nn \l_tmpa_regex { (\ur{l_tmpa_regex}? z) }
\regex_show:N \l_tmpa_regex

shows the following (with characters helpfully explicited, and with the regex inserted in the correct place). Note that \ur{l_tmpa_regex} is treated as a black box, so this does not behave the same as ( a | b | c*?z ), in which z would only appear in one of the branches.

> Compiled regex variable \l_tmpa_regex:
+-branch
  ,-group begin
  | ,-group begin (no capture)
  | | char code 97 (a)
  | +-branch
  | | char code 98 (b)
  | +-branch
  | | char code 99 (c), repeated 0 or more times, greedy
  | `-group end, repeated between 0 and 1 times, greedy
  | char code 122 (z)
  `-group end.