8

The context of this question is that I have a command defined using xparse that slurps in a verbatim argument which can be long. I want to do a replacement on the resulting token list, but the token that I want to replace is the new line character. According to \tl_show:n {#1} this is ^^M. I'm having great difficulty in getting this into \tl_replace_all:Nnn to do the replacement.

If my token list happens to begin with ^^M then I can slurp it from there and use that. So I know that it works if I can just get the ^^M character into a token list. But I don't see how to get it in without this hack.

Here's a minimum sort of working example:

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\cs_generate_variant:Nn \tl_replace_all:Nnn {NV}

\NewDocumentCommand \getstuff {+v}
{
  \tl_set:Nn \l_tmpa_tl {#1}
  \tl_set:Nx \l_tmpb_tl {\tl_head:N \l_tmpa_tl}
  \tl_replace_all:NVn \l_tmpa_tl \l_tmpb_tl {!wibble!}
  \tl_use:N \l_tmpa_tl
}

\ExplSyntaxOff

\begin{document}

\getstuff+
some stuff
+

\getstuff+some stuff+

\end{document}
2
  • 1
    In order to better understand what a token list contains, you can use \tl_show_analysis:n, which requires \usepackage{l3tl-analysis}. You'll see ^^M (the character ^^M) which means ^^M with category code 12.
    – egreg
    Commented Jun 27, 2013 at 23:09
  • @LoopSpace Hi, this is just to call you, in case you want to answer a question of mine tex.stackexchange.com/questions/196498/… with, e.g., calligraphic curved parenthesis from your spath3 package.
    – Manuel
    Commented Sep 21, 2014 at 23:04

2 Answers 2

6

With the help of \tl_analysis_show:n you can get a better result:

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\cs_generate_variant:Nn \tl_replace_all:Nnn {NV}

\str_const:Nx \c_other_eol_str { \cs_to_str:N \^^M }

\NewDocumentCommand \getstuff {+v}
 {
  \tl_analysis_show:n { #1 } % for debugging
  \tl_set:Nn \l_tmpa_tl {#1}
  \tl_replace_all:NVn \l_tmpa_tl \c_other_eol_str {!wibble!}
  \tl_analysis_show:N \l_tmpa_tl % for debugging
  \tl_use:N \l_tmpa_tl
 }

\ExplSyntaxOff

\begin{document}

\getstuff+
some stuff
+

\getstuff+some stuff+

\end{document}

The log has, for the first call of \getstuff,

The token list contains the tokens:
>  ^^M (the character ^^M)
>  s (the character s)
>  o (the character o)
>  m (the character m)
>  e (the character e)
>    (blank space  )
>  s (the character s)
>  t (the character t)
>  u (the character u)
>  f (the character f)
>  f (the character f)
>  ^^M (the character ^^M).

This confirms that ^^M has been absorbed as a token with category code 12; since \c_other_eol_str contains exactly a token with the same property, the next diagnostic message says

The token list \l_tmpa_tl contains the tokens:
>  ! (the character !)
>  w (the letter w)
>  i (the letter i)
>  b (the letter b)
>  b (the letter b)
>  l (the letter l)
>  e (the letter e)
>  ! (the character !)
>  s (the character s)
>  o (the character o)
>  m (the character m)
>  e (the character e)
>    (blank space  )
>  s (the character s)
>  t (the character t)
>  u (the character u)
>  f (the character f)
>  f (the character f)
>  ! (the character !)
>  w (the letter w)
>  i (the letter i)
>  b (the letter b)
>  b (the letter b)
>  l (the letter l)
>  e (the letter e)
>  ! (the character !).

Hurray! The ^^M has been correctly recognized and changed into !wibble!.

A different strategy is by using l3regex:

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand \getstuff {+v}
 {
  \tl_analysis_show:n { #1 } % for debugging
  \tl_set:Nn \l_tmpa_tl {#1}
  \regex_replace_all:nnN { \r } { !wibble! } \l_tmpa_tl
  \tl_analysis_show:N \l_tmpa_tl % for debugging
  \tl_use:N \l_tmpa_tl
 }

\ExplSyntaxOff

\begin{document}

\getstuff+
some stuff
+

\getstuff+some stuff+

\end{document}

In this case !wibble! will be all with category code 12 characters in the modified token list; but of course you don't really want to change ^^M into !wibble!, do you?

3
  • Just to clarify, in my actual case then I wanted to replace ^^M by a macro so, no, !wibble! was not the desired outcome. (To complete the reference, the ! are symbolic of pencils.) Commented Jun 28, 2013 at 12:57
  • @AndrewStacey If you want to replace ^^M with the macro \foo with \regex_replace_all:nnN, say {\c{foo}} in the second argument, instead of {!wibble!}.
    – egreg
    Commented Jun 28, 2013 at 13:43
  • Useful to know. In this case, \tl_replace_all:NVn is sufficient to my needs so I shan't load l3regex. But as always it's good to see examples of these more advanced tools. Commented Jun 28, 2013 at 13:49
6

This seems to work

{\char_set_catcode_other:N \^^M\tl_gset:Nn\g_tmpa_tl{^^M}}
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .