# Retaining spaces in regex replacement across iterations (LaTeX3/l3regex)

I have run into an issue with spaces when using regex replacement. Not entirely sure if this is expansion related or there is some method to properly preserve the spaces.

The current workflow is that I search for [longer] (including brackets) and want to append something after longer (but inside the brackets still - e.g. [longer~and~longer]. Because I never know what that stuff could be (due to multiple iterations of appending), I need to extract the match first, remove the final ], and then append the new stuff plus a replacement ].

This workflow is fine (and appears in the MWE) and retains any space(s) in the original token list. However the replacement token list, when updated into the original token list (via another regex replace) gets all of its spaces gobbled up. What I would expect as [longer~and~longer] ends up as [longerandlonger].

What do I need to add/change in this MWE to retain the spaces from the original regex extraction?

\documentclass{article}

\usepackage{expl3}%
\usepackage{l3regex}%
\usepackage{xstring}%
\usepackage{xparse}%

\ExplSyntaxOn
% Create a new token list to store the node commands
\tl_new:N \g_some_stuff_tl
\tl_gset:Nn \g_some_stuff_tl { This~is~a~[longer]~sentence }

% Create a temporary sequence to store the value of a regex match
\seq_new:N \l_original_match_seq
\seq_clear:N \l_original_match_seq

% Create a temporary token list to store
%and modify the value of a regex match (plus changes)
\tl_new:N \l_update_stuff_tl
\tl_set:Nn \l_update_stuff_tl { }

\cs_generate_variant:Nn \regex_match:nnTF {nVTF}

{%

\regex_match:nVTF { $.*?$ } \g_some_stuff_tl
{%

\cs_generate_variant:Nn \regex_extract_once:nnNF {nVNF}
\regex_extract_once:nVNF { $.*?$ }
\g_some_stuff_tl
\l_original_match_seq
{error}%

%\tl_show_analysis:N - indicates spaces are present in \g_some_stuff_tl
%\seq_show:N - indicates spaces are present in \l_original_match_seq

\seq_pop:NNF \l_original_match_seq \l_update_stuff_tl
{error}%

%\tl_show_analysis:N - indicates spaces are present in \l_update_stuff_tl

\regex_replace_once:nnN { \]$} { } \l_update_stuff_tl %\tl_show_analysis:N - indicates spaces are still present in \l_update_stuff_tl \tl_put_right:Nn \l_update_stuff_tl { ~and~longer] } %\tl_show_analysis:N - indicates spaces are still present in \l_update_stuff_tl \cs_generate_variant:Nn \regex_replace_all:nnN {nVN} \regex_replace_once:nVN { $.*?$ } \l_update_stuff_tl \g_some_stuff_tl %\tl_show_analysis:N - spaces in the replacement text have disappeared %from \g_some_stuff_tl - but still exist elsewhere } {error} }% \ExplSyntaxOff \begin{document} \addsomestuff \addsomestuff \addsomestuff \end{document}  I found a workaround for this problem... but it is a hack that I think is the wrong approach. Before the final \regex_replace_once:nVN I can do a replace all on the updated token list (\l_update_stuff_tl) converting \s to something else which is unlikely to appear every (such as ~!@#$%^&()). Then, following the final \regex_replace_once:nVN I do another replace all on that unlikely token list by replacing them with \.

While this hack works, I am looking for more correct way to handle this. This workaround also has the drawback that a user, however unlikely, may actually enter the string and it would cause a probable error or lots of confusion.

Edit: Changed \l_some_stuff_tl to \g_some_stuff_tl because setting its value should persist outside the command.

• Not the main point, but variants should not be generated in the body of the macro. – egreg Mar 15 '17 at 7:30
• @egreg Good call. This is mostly habit from a programming background where definitions in the same scope are preferred and global access is bad, unless necessary. I updated this in the code above for future viewers not to pick up this habit. – whatisit Mar 15 '17 at 22:07

You can “remember” a match; also, you can use the contents of a token list variable in the replacement text with \u{<tl name>}.

\documentclass{article}

\usepackage{xparse}
\usepackage{l3regex}

\ExplSyntaxOn
% Create a new token list to store the node commands
\tl_new:N \g_whatisit_some_stuff_tl
\tl_gset:Nn \g_whatisit_some_stuff_tl {a~[longer]~sentence}

% Create a temporary token list to store
% and modify the value of a regex match (plus changes)
\tl_new:N \l_whatisit_update_stuff_tl
\tl_set:Nn \l_whatisit_update_stuff_tl { ~and~longer }

% Temporary storage
\tl_new:N \l_whatisit_temporary_tl

\cs_generate_variant:Nn \regex_match:nnTF {nVTF}

{
\regex_match:nVTF { $.*?$ } \g_some_stuff_tl
{
\tl_set_eq:NN \l_whatisit_temporary_tl \g_whatisit_some_stuff_tl
\regex_replace_once:nnN { ($.*?)$ } { \1\u{l_update_stuff_tl}\] }
\l_whatisit_temporary_tl
\tl_gset_eq:NN \g_whatisit_some_stuff_tl \l_whatisit_temporary_tl
}
{error}
\tl_use:N \g_whatisit_some_stuff_tl
}
\ExplSyntaxOff

\begin{document}


Since the variable you act on is global and \regex_replace_once:nnN only acts locally, you have to use a temporary local variable
• Got it, thanks. I had seen the \0, \1, \2, etc. mentioned in the documentation, but didn't quite follow that they refer to the ( ... ) groupings in the regex. Understanding this makes it much easier. This is also great because it cuts out all the unnecessary sequence-related code! I followed you on the global variable / temporary variable stuff and updated my package using it without issue. – whatisit Mar 15 '17 at 22:04