12

I would like to use LaTeX3 regular expressions to extract the "lines" from text like the following:

- First line
- second line ending with \LaTeX
- third line with - in the middle
- fourth line that
goes on for a bit

By a "line" I mean text that starts with a dash - that occurs at the beginning of a "physical line", and then continues until the next such dash, or the end of the string.

My first guess was that something like the following would work:

\regex_split:nVN {  [\f\n\r]\s+-\s+  } { \Lines } \l_tmp_seq

My understanding is that this should match a "newline character" followed by - that is surrounded by white-space on both sides. Unfortunately, this fails horribly, producing only one match:

enter image description here

As this didn't work I thought that I would try a very simple-minded approach and just split the string on white-space surrounded dashes -. This would give the wrong result on lines like the third line above, but I can use -- in the middle of the line so this is a workable alternative. As \s matches white space, the following should split lines on -:

\regex_split:nVN { \s+-\s+  } { \Lines } \l_tmp_seq

This is better, but the results are still not perfect because, in addition to the expected problem on line 3, having \LateX at the end of a line confuses \regex_split presumably because macros tends to swallow spaces:

enter image description here

The best solution that I have found is using:

\regex_split:nVN { \B-\B } { \Lines } \l_tmp_seq

This produces the output that I was expecting from the last regex:

enter image description here

That is, we are getting an unwanted match for the - in the middle of the third line but this is otherwise OK.

Question Is there a regular expression that does not return any false matches? Initial and/or final matching empty sequences are allowed.

[Aside: I am not worried about the "empty" initial sequences above because I know that can get rid of them using

\seq_pop_left:NN \l_tmp_seq \l_tmpa_tl % pop initial blank

Terminating empty sequences can, of course, be dealt with in almost exactly the same way using \seq_pop_right:NN.]

Here is a full MWE to produce the examples above:

\documentclass{article}
\usepackage{xparse}
\parindent=0pt

\newcommand\BS{$\backslash$}
\newcommand\Test[1]{\textbf{Testing:}\quad\texttt{#1}}
\begin{document}

\newcommand\Lines{
    - First line
    - second line ending with \LaTeX
    - third line with -- in the middle
    - fourth line that
    goes on for a bit
}

\ExplSyntaxOn
\seq_new:N \l_tmp_seq
\cs_generate_variant:Nn \regex_split:nnN { nVN }

\Test{[\BS f\BS n\BS r]\BS s+-\BS s+}
\regex_split:nVN { [\f\n\r]\s+-\s+  } { \Lines } \l_tmp_seq
\begin{itemize}\item\seq_use:Nn \l_tmp_seq {\item }\end{itemize}

\Test{\BS s-\BS s}
\regex_split:nVN { \s-\s  } { \Lines } \l_tmp_seq
\begin{itemize}\item\seq_use:Nn \l_tmp_seq {\item }\end{itemize}

\Test{\BS B-\BS B}
\regex_split:nVN { \B-\B  } { \Lines } \l_tmp_seq
\begin{itemize}\item\seq_use:Nn \l_tmp_seq {\item }\end{itemize}

\ExplSyntaxOff

\end{document}

Escaping the - in the regular expressions doesn't help either.

Finally, the problem may be that the newline characters in \Lines are stripped when it is defined. As far as I can see this is OK because the code above gives similar output. In particular, if I use:

\documentclass{article}
\usepackage{xparse}
\parindent=0pt

\begingroup\obeylines
\gdef\Lines{
    - First line
    - second line ending with \LaTeX
    - third line with -- in the middle
    - third line that
    goes on for a bit
}
\endgroup

\begin{document}

  \ExplSyntaxOn
  \seq_new:N \l_tmp_seq
  \cs_generate_variant:Nn \regex_split:nnN { nVN }
  \regex_split:nVN { [\f\n\r]\s+\-\s+  } { \Lines } \l_tmp_seq

  Sequence~length:~ \seq_count:N \l_tmp_seq.
  \begin{itemize}\item\seq_use:Nn \l_tmp_seq {\item }\end{itemize}
  \ExplSyntaxOff

\end{document}

then I get the output:

enter image description here

In practice, I am getting \Lines as the \BODY of a environ environment, so I don't really have the luxury of this definition...

  • 1
    The problem is certainly that newline chararacters are converted to spaces, and with \obeylines I get correct splitted results. – Ulrike Fischer Oct 30 '17 at 11:54
  • @UlrikeFischer Looks like an answer to me! – Joseph Wright Oct 30 '17 at 12:02
  • @Andrew: well I'm sure that \obeylines or something equivalent is needed, I'm not sure that your regex is completly correct but will check again later. – Ulrike Fischer Oct 30 '17 at 12:11
  • @Andrew To TeX's eyes, abc<newline>def and abc def are exactly the same after the input has been tokenized, assuming standard category codes. – egreg Oct 30 '17 at 12:46
10

You need obeylines or something equivalent so that the newline char doesn't disappear. The following splits the lines (nearly) as wanted. Instead of \^^M one can also use \r (I only used the first because \show\Lines showed ^^M).

This creates this sequence:

The sequence \l_tmp_seq contains the items (without outer braces):
>  {First line}
>  {second line ending with \LaTeX }
>  {third line with -- in the middle}
>  {fourth line that^^Mgoes on for a bit^^M}.
<recently read> }

One problem could be that the fourth item contains a ^^M and so a probably unwanted line break.

\documentclass{article}
\usepackage{xparse}
\parindent=0pt

\newcommand\BS{$\backslash$}
\newcommand\Test[1]{\textbf{Testing:}\quad\texttt{#1}}
\begin{document}

\begingroup
\obeylines
\gdef\Lines{
    - First line
    - second line ending with \LaTeX
    - third line with -- in the middle
    - fourth line that
    goes on for a bit
}

\endgroup

\ExplSyntaxOn
\seq_new:N \l_tmp_seq
\cs_generate_variant:Nn \regex_split:nnN { nVN }

\Test{}\verb+^^M\-+
\regex_replace_once:nnN { \A\^^M?\-\s* } {} \Lines %to get rid of the first ^^M
\regex_split:nVN { \^^M\-\s* } \Lines \l_tmp_seq

\begin{itemize}\item\seq_use:Nn \l_tmp_seq {\item }\end{itemize}


\ExplSyntaxOff

\end{document}

enter image description here

  • 1
    You can remove the initial hyphen and the possible ^^M by first doing \regex_replace_once:nnN { \A\^^M?\-\s* } {} \Lines; for the splitting you can do \regex_split:nVN { \^^M\-\s* } \Lines \l_tmp_seq and also the initial space will vanish. Note that \Lines should not be braced: the argument type V denotes a single token. – egreg Oct 30 '17 at 13:37
  • 1
    @Andrew TeX works with tokens; during tokenization, end-of-lines are discarded and (usually) substituted with something else (normally a space). – egreg Oct 30 '17 at 13:39
  • ^^M, ^^J are tex specialities, ^^M is ASCII 13 (return). – Ulrike Fischer Oct 30 '17 at 13:41
  • 2
    @UlrikeFischer By the way, you can use \r instead of \^^M in the search regex. – egreg Oct 30 '17 at 13:46
  • Is there a description somewhere of ^^M, ^^J, ^^K, etc? I just did a quick google search and didn't find anything. – Andrew Oct 30 '17 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.