Building regex from another one

I'm trying to define a command where the user can define a regex, which should then be used to create a larger regex that in turn will be matched on a token list.

The naive approach to insert it from a token list obviously fails because of a wrong catcode setup:

\documentclass{article}
\usepackage{expl3}

\begin{document}
\ExplSyntaxOn

\tl_set:Nn \l_foo_tl { [a-z]+ }
\regex_const:Nn \l_foo_regex { (\w+)( $\u{l_foo_tl}$ ) }
\regex_show:N \l_foo_regex

\seq_new:N \l_foo_seq
\regex_extract_all:NnN \l_foo_regex { a[x], b[yy], c[zzz] } \l_foo_seq
\seq_show:N \l_foo_seq

\ExplSyntaxOff
\end{document}


outputs

+-branch
,-group begin
| Match, repeated 1 or more times, greedy
|   range [97,122]
|   range [65,90]
|   range [48,57]
|   char code 95
-group end
,-group begin
| char code 91
| char 91, catcode 12
| char 97, catcode 11
| char 45, catcode 12
| char 122, catcode 11
| char 93, catcode 12
| char 43, catcode 12
| char code 93
-group end.


What I'd want is either something like

\regex_const:Nn \l_sub_regex { [a-z]+ }
\regex_const:Nn \l_foo_regex { (\w+)( $...$ ) }


where ... somehow inserts the regex represented by \l_sub_regex (both \c{l_sub_regex} and \u{l_sub_regex} give wrong results here); or a way to convert a compiled regex back to its string representation, something like \regex_to_str:N.

Perhaps there's a way to insert it back from a token list using some \detokenize or \scantokens hackery, but I'm wondering if l3regex already provides a proper solution for this.

EDIT: I found a note in the l3regex documentation about features that are "likely to be implemented at some point in the future":

Provide a syntax such as \ur{l_my_regex} to use an already-compiled regex in a more complicated regex. This makes regexes more easily composable.

So it seems such a feature doesn't currently exist but is planned for the future.

(By the way, it would be really helpful if the \regex_show: functions would also print the actual ASCII representation of a character if it is in the set of printable characters. Several lines of char code XXX are harder to debug than necessary.)

EDIT 2: Unfortunately the accepted answer doesn't seems to work correctly when a complete regex is inserted in a capturing group. The following code reproduces the problem:

\regex_set:Nn \l_tmpa_regex { a|b|c }
\cs_show:N \l_tmpa_regex

\regex_set:Nn \l_tmpa_regex { (\y{l_tmpa_regex}) }
\regex_set:Nn \l_tmpb_regex { (a|b|c) }
\regex_show:N \l_tmpa_regex
\regex_show:N \l_tmpb_regex
\cs_show:N \l_tmpa_regex
\cs_show:N \l_tmpb_regex

\regex_extract_once:NnNTF \l_tmpa_regex {a} \l_tmpa_seq {} {}
\seq_show:N \l_tmpa_seq
\regex_extract_once:NnNTF \l_tmpb_regex {a} \l_tmpa_seq {} {}
\seq_show:N \l_tmpa_seq


Both regexes are supposed to match on the token list a but only the second one does.

The final regexes in \l_tmpa_regex and \l_tmpb_regex should be the same, as the two identical \regex_show:N outputs suggest:

> Compiled regex variable \l_tmpa_regex:
+-branch
,-group begin
| char code 97 (a)
+-branch
| char code 98 (b)
+-branch
| char code 99 (c)
-group end.

> Compiled regex variable \l_tmpb_regex:
+-branch
,-group begin
| char code 97 (a)
+-branch
| char code 98 (b)
+-branch
| char code 99 (c)
-group end.


But the raw internal structure reveals that there is a difference (code shortened a bit):

> \l_tmpa_regex=macro:->
\__regex_branch:n {
\__regex_group:nnnN {
\__regex_branch:n {
<char a>
\__regex_branch:n { <char b> }
\__regex_branch:n { <char c> }
}
}{1}{0}\c_false_bool
}

> \l_tmpb_regex=macro:->
\__regex_branch:n {
\__regex_group:nnnN {
\__regex_branch:n { <char a> }
\__regex_branch:n { <char b> }
\__regex_branch:n { <char c> }
}{1}{0}\c_false_bool
}


If we look at the originally inserted regex, the internal structure seems fine at the start:

> \l_tmpa_regex=macro:->
\__regex_branch:n { <char a> }
\__regex_branch:n { <char b> }
\__regex_branch:n { <char c> }


Especially because the number of \__regex_branch:n commands is the same in both result regexes, I don't think this is a limitation of the internal regex structure but of the way the accepted answer inserts one regex into another. It's probably related to brace tokens going to the wrong place.

The problem seems to be more compilcated than just shifting some braces around. When l3regex reads the definition of a branch, it has internally already pushed the sequence \__regex_branch:n { \if_false: } \fi: on the result token list. A proper fix would therefore have to check in what context it occurs (beginning/middle/end of a branch) and to modify the internal token list such that the regex to be inserted fits in correctly.

• \def\insertmyregex#1{\regex_const:Nn \l_foo_regex { (\w+)( $#1$ ) }} \insertmyregex{ [a-z]+ } ? – David Carlisle Jun 8 '19 at 15:35
• I'd say it's not possible because the regex module ensures that control sequences don't expand to weird stuff, so it's most certainly by design that the text inserted by \u is matched literally, and not parsed again. I dare say it's for the best. You'd need to manually expand \l_foo_tl before \regex_const:Nn compiles the regex (like David just suggested, for example). (And I agree with your statement about the output of \regex_show:N. Flie an issue! :-) – Phelype Oleinik Jun 8 '19 at 15:43
• @DavidCarlisle That's an easy fix for the simple use case I posted, but it doesn't seem to scale well if you need to combine several regexes that are defined at different locations in the code. So if anyone can see a more general solution, I'd prefer this one. – siracusa Jun 9 '19 at 5:26

After a crash course on l3regex I managed to pull off two (rather stupid) ways to do what you ask. I'll post them as separate answers because they are completely different both at implementation and usage levels.

Disclaimer: This answer uses internal code of l3regex which definitely should not be used in a document so it might break as soon as something changes in the code of l3regex, so please do not use this code unless you really know what you are doing.

Inserting a token list to be compiled as a regex:

This one steps in in the compilation of a regex right after the regular expression is tokenized but before the actual compilation takes place, replacing all \y{<tl var>} by the contents of \<tl var>. It's a really nasty brute-forcing of a token list inside a regular expression, but it apparently works fine.

With this method you can insert any token list to be compiled as a regular expression using \y{<tl var>}:

\tl_const:Nn \c_foo_tl { a-z }
\tl_const:Nn \c_bar_tl { [\y{c_foo_tl}]+ }
\regex_const:Nn \c_foo_regex { (\w+)( $\y{c_bar_tl}$ ) }


In the example above the token list \c_bar_tl is inserted into the regular expression. \c_bar_tl itself contains a recursive call to \y{c_foo_tl}. I offer two different macros which which you can choose if you want to allow recursion or not. If you don't allow recursion, the code above matches the literal string y{c_foo_tl}. If you allow recursion, then \y{c_foo_tl} expands and inserts a-z into the expression, which is then properly compiled. Mind you that with recursion enabled, infinite recursion is also enabled, if your code does so.

I used \y instead of \ur in this one just to keep the same character as in the other answer, but it could easily be \ur or some other string.

I injnected a macro in \__regex_escape_use:nnnn which, right before the regular expression is compiled, replaces all \y{<tl var>} by the contents of \<tl var>. This requires adding a line to \__regex_escape_use:nnnn, from which the code starts. The function then proceeds doing the proper replacements (recursively or not) until it finds the end of the expression. When it does the function returns the control to the regular expression engine to do its job normally.

Because this does the expansion of the escape sequence prior to the compilation of the regular expression there are no bounds on the contents of each \<tl var> used as long as the final expression, after the expansion of all \y results in a valid regular expression. This allows you to define one token list c_foo_tl with [a- an another with \y{c_foo_tl}z]+ and it would expand to the expected [a-z]+.

The function which controls the behaviour after the expansion of an escape sequence \y is \__regex_prescan_yank_eval_continue:n. Below there are two different versions to choose from, recursive and non-recursive, as discussed earlier.

Here's the code:

\documentclass{article}
\usepackage{expl3}
\ExplSyntaxOn
\cs_gset_protected:Npn \__regex_escape_use:nnnn #1#2#3#4
{
\group_begin:
\tl_clear:N \l__regex_internal_a_tl
\cs_set:Npn \__regex_escape_unescaped:N ##1 { #1 }
\cs_set:Npn \__regex_escape_escaped:N ##1 { #2 }
\cs_set:Npn \__regex_escape_raw:N ##1 { #3 }
\__regex_standard_escapechar:
\tl_gset:Nx \g__regex_internal_tl
{ \__kernel_str_to_other_fast:n {#4} }
\tl_put_right:Nx \l__regex_internal_a_tl
{
\exp_after:wN \__regex_escape_loop:N \g__regex_internal_tl
{ break } \prg_break_point:
}
\exp_after:wN
\group_end:
\l__regex_internal_a_tl
}
\cs_new_protected:Npn \__regex_prescan_yank:N #1
{ \tl_set:Nx #1 { \exp_args:NV \__regex_prescan_yank:n #1 } }
\cs_set:Npn \__regex_tmp:w #1#2
{
\cs_new:Npn \__regex_prescan_yank:n ##1
{ \__regex_prescan_yank:w ##1 #1 \q_nil #2 \q_stop }
\cs_new:Npn \__regex_prescan_yank:w ##1 #1 ##2 #2
{
##1
\quark_if_nil:nT {##2}
{ \use_none_delimit_by_q_stop:w }
\__regex_prescan_yank_eval_continue:n {##2}
}
}
\exp_args:Nxx \__regex_tmp:w
{ \__kernel_str_to_other_fast:n { \y } \c_left_brace_str }
{ \c_right_brace_str }
% \cs_new:Npn \__regex_prescan_yank_eval_continue:n #1
%   { % Non-recursive
%     \exp_args:Nv \__kernel_str_to_other_fast:n { #1 }
%     \__regex_prescan_yank:w
%   }
\cs_new:Npn \__regex_prescan_yank_eval_continue:n #1
{ % Recursive
\exp_last_unbraced:Nf \__regex_prescan_yank:w
\exp_args:Nv \__kernel_str_to_other_fast:n { #1 }
}
\ExplSyntaxOff
\begin{document}
\ExplSyntaxOn
\tl_const:Nn \c_foo_tl { a-z }
\tl_const:Nn \c_bar_tl { [\y{c_foo_tl}]+ }
\regex_const:Nn \c_foo_regex { (\w+)( $\y{c_bar_tl}$ ) }
% \regex_const:Nn \c_foo_regex { (\w+)( $[a-z]+$ ) }
\regex_show:N \c_foo_regex

\seq_new:N \l_foo_seq
\regex_extract_all:NnN \c_foo_regex { a[x], b[yy], c[zzz] } \l_foo_seq
\seq_show:N \l_foo_seq
\ExplSyntaxOff
\end{document}


After a crash course on l3regex I managed to pull off two (rather stupid) ways to do what you ask. I'll post them as separate answers because they are completely different both at implementation and usage levels.

Disclaimer: This answer uses internal code of l3regex which definitely should not be used in a document so it might break as soon as something changes in the code of l3regex, so please do not use this code unless you really know what you are doing.

Inserting a compiled regex into another regex:

This one goes sort of along the lines of what you found in the documentation:

Provide a syntax such as \ur{l_my_regex} to use an already-compiled regex in a more complicated regex. This makes regexes more easily composable.

You first need to compile a regex using \regex_const:Nn or \regex:set:Nn or something like that, and then use the compiled regex inside the new regular expression:

\regex_const:Nn \c_bar_regex { [a-z]+ } % Compile \c_bar_regex
\regex_const:Nn \c_foo_regex { (\w+)( $\y{c_bar_regex}$ ) } % Insert it with \y{...}


I used \y instead of \ur because it wouldn't require too much changes in the l3regex code, instead I just needed to define a \__regex_compile_/y: function. It is the "cleanest" method because it doesn't require to change any of the internals of l3regex (though it uses a handful of them).

I defined an escape sequence \y{<regex var>}, much similar to \u{<tl var>} (in fact, most of the code is a copy replacing u by y), with the difference that \u expands \<tl var> and inserts the literal tokens inside it to be matched (exactly as in the question), while \y fetches the contents of the \<regex var> and injects it in the current regular expression being compiled.

This method seems the "cleverest" one, but it has major limitations. Since each regex is compiled in advance, they need to be a full regular expression, otherwise either the compilation will fail or the regex will mean something different. For instance, take the regular expression [a-z]+ and split it into two sub expressions: A=a-z, and B=[A]+. The complete expression matches the range [a,z], repeated 1 or more times, greedy. The sub-expression a-z, however, matches the three literal characters a-z, since - doesn't have the "range" meaning outside of [...]. Once you put A in B, B will match a, -, or z, repeated 1 or more times, greedy.

The usage of \y inside [...] can be disabled to prevent this type of problem, just un-comment the commented lines in \__regex_compile_/y:.

Reply to Edit 2: The initial version of the code would break in more complicated cases, as you noted. I stand my opinion that I prefer the approach in the other answer since it requires no knowledge of the inner workings of l3regex (which is about my level of knowledge :-), while this one messes with the internal structure of a compiled regex.

When l3regex compiles a regex of the form (a|b), the underlying regex is something like branch { group { a-branch b-branch } }, however when you add a token list with \u (code upon which the \y is built upon) the engine adds another branch inside the group: branch { group { branch { whatever-is-in-\u{...} } } }, and this additional branch is in an upper layer of the code which can't be changed by just adding an escape sequence like this code does. Luckily the l3regex doesn't seem to mind that added branch when matching the regex, so the output of:

% result of compiling "(a|b|c)"
branch { group { a-branch b-branch c-branch } }


and

% result of compiling "(\y{l_tmpa_tl})" with \l_tmpa_tl=a|b|c
branch { group { branch { a-branch b-branch c-branch } } }


seem to be the same (with a rather limited set of tests). So a small tweak to the previous code to take care of multiple branches seem to do the trick. If the code refuses to work with the added branch then a deeper mess-up of l3regex's code will probably be necessary to fix this. Let me know if something won't work right.

Here's the code:

\documentclass{article}
\usepackage{expl3}
\ExplSyntaxOn
\cs_new_protected:cpn { __regex_compile_/y: } #1#2
{
% Disable "dangerous" usage in [...]
% \__regex_if_in_class_or_catcode:TF
%   { \__regex_compile_raw_error:N y #1 #2 }
%   {
\__regex_two_if_eq:NNNNTF #1 #2 \__regex_compile_special:N \c_left_brace_str
{
\tl_set:Nx \l__regex_internal_a_tl { \if_false: } \fi:
\__regex_compile_y_loop:NN
}
{
\__kernel_msg_error:nn { kernel } { u-missing-lbrace }
\__regex_compile_raw:N y #1 #2
}
% }
}
\cs_new:Npn \__regex_compile_y_loop:NN #1#2
{
\token_if_eq_meaning:NNTF #1 \__regex_compile_raw:N
{ #2 \__regex_compile_y_loop:NN }
{
\token_if_eq_meaning:NNTF #1 \__regex_compile_special:N
{
\exp_after:wN \token_if_eq_charcode:NNTF \c_right_brace_str #2
{ \if_false: { \fi: } \__regex_compile_y_end: }
{ #2 \__regex_compile_y_loop:NN }
}
{
\if_false: { \fi: }
\__kernel_msg_error:nnx { kernel } { u-missing-rbrace } {#2}
\__regex_compile_y_end:
#1 #2
}
}
}
\cs_new_protected:Npn \__regex_compile_y_end:
{
\tl_set:Nv \l__regex_internal_a_tl { \l__regex_internal_a_tl }
\exp_args:NV \__regex_analyse_y:n \l__regex_internal_a_tl
}
\cs_new_protected:Npn \__regex_analyse_y:n #1
{
{
\reverse_if:N \if_meaning:w \__regex_branch:n #1
\msg_error:nn { siracusa / regex } { unknown-condition }
\else:
\exp_args:No \__regex_analyse_y_aux:n { \use_none:n #1 }
\fi:
}
{ \msg_error:nn { siracusa / regex } { unknown-condition } }
}
\cs_new_protected:Npn \__regex_analyse_y_aux:n #1
{
\tl_if_empty:oTF { \use_none:n #1 }
{ \exp_args:NNo \tl_build_put_right:Nn \l__regex_build_tl { \use:n #1 } }
{ \tl_build_put_right:Nn \l__regex_build_tl { \__regex_branch:n #1 } }
}
\msg_new:nnn { siracusa / regex } { unknown-condition }
{ Unknown/unimplemented~condition~in~code. }
\ExplSyntaxOff
\begin{document}
\ExplSyntaxOn
\regex_const:Nn \c_bar_regex { [a-z]+ }
\regex_const:Nn \c_foo_regex { (\w+)( $\y{c_bar_regex}$ ) }
% \regex_const:Nn \c_foo_regex { (\w+)( $[a-z]+$ ) }
\regex_show:N \c_foo_regex

\seq_new:N \l_foo_seq
\regex_extract_all:NnN \c_foo_regex { a[x], b[yy], c[zzz] } \l_foo_seq
\seq_show:N \l_foo_seq
\ExplSyntaxOff
\end{document}


Bonus, for thee who disliketh the output of \regex_show

You need to patch \__regex_show:N:

\ExplSyntaxOn
\cs_new:Npn \__regex_show_if_visible_ascii:n #1
{
\if_int_compare:w
\if_int_compare:w \int_eval:n{#1}>31  ~ 1 \else: 0 \fi:
\if_int_compare:w \int_eval:n{#1}<127 ~ 1 \else: 0 \fi:
= 11 \exp_stop_f:
\c_space_tl (\char_generate:nn {#1} {12})
\fi:
}
\cs_set_protected:Npn \__regex_show:N #1
{
\group_begin:
\tl_build_begin:N \l__regex_build_tl
\cs_set_protected:Npn \__regex_branch:n
{
\seq_pop_right:NN \l__regex_show_prefix_seq
\l__regex_internal_a_tl
\__regex_show_one:n { +-branch }
\seq_put_right:No \l__regex_show_prefix_seq
\l__regex_internal_a_tl
\use:n
}
\cs_set_protected:Npn \__regex_group:nnnN
{ \__regex_show_group_aux:nnnnN { } }
\cs_set_protected:Npn \__regex_group_no_capture:nnnN
{ \__regex_show_group_aux:nnnnN { ~(no~capture) } }
\cs_set_protected:Npn \__regex_group_resetting:nnnN
{ \__regex_show_group_aux:nnnnN { ~(resetting) } }
\cs_set_eq:NN \__regex_class:NnnnN \__regex_show_class:NnnnN
\cs_set_protected:Npn \__regex_command_K:
{ \__regex_show_one:n { reset~match~start~(\iow_char:N\\K) } }
\cs_set_protected:Npn \__regex_assertion:Nn ##1##2
{
\__regex_show_one:n
{ \bool_if:NF ##1 { negative~ } assertion:~##2 }
}
\cs_set:Npn \__regex_b_test: { word~boundary }
\cs_set_eq:NN \__regex_anchor:N \__regex_show_anchor_to_str:N
\cs_set_protected:Npn \__regex_item_caseful_equal:n ##1
{
\__regex_show_one:n
{
char~code~\int_eval:n{##1}
}
}
\cs_set_protected:Npn \__regex_item_caseful_range:nn ##1##2
{
\__regex_show_one:n
{
range~[
\int_eval:n{##1} \__regex_show_if_visible_ascii:n {##1}, % <-- Added
\int_eval:n{##2} \__regex_show_if_visible_ascii:n {##2}  % <-- Added
]
}
}
\cs_set_protected:Npn \__regex_item_caseless_equal:n ##1
{
\__regex_show_one:n
{
char~code~\int_eval:n{##1}
}
}
\cs_set_protected:Npn \__regex_item_caseless_range:nn ##1##2
{
\__regex_show_one:n
{
Range~[
\int_eval:n{##1} \__regex_show_if_visible_ascii:n {##1}, % <-- Added
\int_eval:n{##2} \__regex_show_if_visible_ascii:n {##2}  % <-- Added
]~(caseless)
}
}
\cs_set_protected:Npn \__regex_item_catcode:nT
{ \__regex_show_item_catcode:NnT \c_true_bool }
\cs_set_protected:Npn \__regex_item_catcode_reverse:nT
{ \__regex_show_item_catcode:NnT \c_false_bool }
\cs_set_protected:Npn \__regex_item_reverse:n
{ \__regex_show_scope:nn { Reversed~match } }
\cs_set_protected:Npn \__regex_item_exact:nn ##1##2
{
\__regex_show_one:n
{
char~##2
,~catcode~##1
}
}
\cs_set_eq:NN \__regex_item_exact_cs:n \__regex_show_item_exact_cs:n
\cs_set_protected:Npn \__regex_item_cs:n
{ \__regex_show_scope:nn { control~sequence } }
\cs_set:cpn { __regex_prop_.: } { \__regex_show_one:n { any~token } }
\seq_clear:N \l__regex_show_prefix_seq
\__regex_show_push:n { ~ }
\cs_if_exist_use:N #1
\tl_build_end:N \l__regex_build_tl
\exp_args:NNNo
\group_end:
\tl_set:Nn \l__regex_internal_a_tl { \l__regex_build_tl }
}
\ExplSyntaxOff


then \regex_show:n { (\w+)( $[a-z]+$ ) } will print:

+-branch
,-group begin
| Match, repeated 1 or more times, greedy
|   range [97 (a),122 (z)]
|   range [65 (A),90 (Z)]
|   range [48 (0),57 (9)]
|   char code 95 (_)
-group end
,-group begin
| char code 91 ([)
| range [97 (a),122 (z)], repeated 1 or more times, greedy
| char code 93 (])
-group end.

• Excellent! That's actually the same idea I had in mind for the implementation, but you beat me to it. :) Is there any chance this would go into the l3regex package, perhaps as command \R instead? – siracusa Jun 11 '19 at 20:38
• @siracusa Personally, I prefer the other option (seems more robust and more flexible to me :-) I'm not in the L3 team (yet :) so I can't really say if it will or will not be implemented. I raised it in the issue tracker: github.com/latex3/latex3/issues/590. Let's see what is made of that. – Phelype Oleinik Jun 12 '19 at 9:44
• @siracusa I added something to improve the output of \regex_show. – Phelype Oleinik Jun 12 '19 at 9:52
• Thanks for posting that issue. What I like about this \y implementation is that it's more efficent if you want to work with an already compiled regex or need both regexes in compiled form in your program. Maybe it would even be useful to have both these features in the package implementation. I can see use cases for either of them. – siracusa Jun 12 '19 at 15:38
• Could you please have a look at my last edit to the question? There seems to be a problem when inserting a complete regex into a capturing group. I'm quite sure this is no fundamental problem but just comes from one of the inserted braces going to the wrong place. – siracusa Jul 12 '19 at 11:38