# How do use {<setup>} in \tl_set_rescan:Nnn to replace \scantokens?

This question is related to (Changing escape character in \tl_set_rescan:Nnn) and (https://tex.stackexchange.com/a/496258/7832) but is not the same.

I've tried to change this:

    \exp_args:NNx \_scontents_append_contents:nn \l_scontents_name_seq_tl
{
\exp_not:N \scantokens \exp_after:wN { \l_tmpa_tl } %
}


for something else in the expl3 way using \tl_set_rescan:Nnn, like this:

    \exp_args:NNx \_scontents_append_contents:nn \l_scontents_name_seq_tl
{
< use \tl_set_rescan:Nnn over \l_tmpa_tl to obtain the same effect >
}


But I can't get the same effect, I don't know what to put in {<setup>} to get the desired result. The MWE file:

\documentclass{article}
\usepackage{filecontentsdef}[2019/04/20]
\usepackage{xparse}
\usepackage{fvextra}
\setlength{\parindent}{0pt} % just for the example
\ExplSyntaxOn

\keys_define:nn { scontents }
{
save-env  .tl_set:N   = \l_scontents_name_seq_tl,
save-env  .initial:n  = contents,
show-env  .bool_set:N = \l_scontents_show_env_bool,
show-env  .initial:n  = false
}

\cs_new_protected:Npn \_scontents_append_contents:nn #1 #2
{
\seq_if_exist:cF { g_scontents_seq_name_#1_seq }
{
\seq_new:c { g_scontents_seq_name_#1_seq }
}
\seq_gput_right:cn { g_scontents_seq_name_#1_seq } { #2 }
}

\cs_new_protected:Npn \_scontents_getfrom_seq:nn #1 #2
{
\seq_item:cn { g_scontents_seq_name_#2_seq } { #1 }
}

\ProvideExpandableDocumentCommand{\getstored}{ O{1} m }
{
\_scontents_getfrom_seq:nn { #1 } { #2 }
}

% Define scontents (wrap \filecontentsdefmacro) whit [key=val] Delaying
\ProvideDocumentEnvironment{ scontents }{}
{
\char_set_catcode_active:N \^^M
\scontents_start_environment:w
}
{
\scontents_stop_environment:
\scontents_atend_environment:
}

\cs_new_protected:Npn \scontents_environment_keys:w [#1]
{
\keys_set:nn { scontents } { #1 }
}

% Star environment
\group_begin:
\char_set_catcode_active:N \^^M
\cs_new_protected:Npn \scontents_start_environment:w #1 ^^M
{
\tl_if_blank:nF { #1 } { \scontents_environment_keys:w #1 }
\group_begin: % open group for env
\use:c { filecontentsdefmacro } { \l_tmpa_tl } ^^M
}
\group_end:

% Stop environment
\cs_new_protected:Nn \scontents_stop_environment:
{
\endfilecontentsdefmacro
\group_end:  % close group for env
}

% A variant to replace \^^M for \^^J (need by Verb{..} from fvextra)
% https://tex.stackexchange.com/a/8971/7832
\cs_generate_variant:Nn \tl_replace_all:Nnn { Nxx }

% Expand \l_tmpa_tl and pass to seq
\cs_gset_protected:Nn \_scontents_macro_to_seq:
{
\regex_replace_all:nnN { \^^M } { \^^J } \l_tmpa_tl
\cs_log:N \l_tmpa_tl
\exp_args:NNx \_scontents_append_contents:nn \l_scontents_name_seq_tl
{
\exp_not:N \scantokens \exp_after:wN { \l_tmpa_tl } %
}
}

% Code after scontent environment
\cs_new_protected:Nn \scontents_atend_environment:
{
\_scontents_macro_to_seq:
\bool_if:NT \l_scontents_show_env_bool
{
\_scontents_getfrom_seq:nn { -1 }{ \l_scontents_name_seq_tl }
}
\cs_undefine:N \l_tmpa_tl
}
\ExplSyntaxOff
\begin{document}
\section*{Test environment}
Test \verb+\begin{scontents}+ no \verb+[key=val]+\par

\begin{scontents}
Using \verb+scontents+ env no \verb+[key=val]+, save in \verb+contents+
with index $1$\footnote{AND footnotes !!}.%

Prove new \Verb*{ new fvextra whit braces } and environment \verb+Verbatim+

\begin{Verbatim}
(A) verbatim environment                                               %
many space here and percent at end of line                             %
many space
\end{Verbatim}
No espace after, see \verb+https://tex.stackexchange.com/a/460967/7832+.\par
Use \verb|\relax|.\relax
\end{scontents}

Test \verb+\begin{scontents}[save-env=other]+\par

\begin{scontents}[save-env=other]
Now Using \verb+scontents+ env with \verb+[save-env=other]+, save in
\verb+other+ with index $1$.

\begin{verbatim*}
(B) verbatim environment

kkkkkkkkkkkkkkkkkkkkkkkk
\end{verbatim*}
Show a space.
\end{scontents}

\section*{Show stored contents}
XX\getstored[1]{contents}NO space here :)\par

YY\getstored[1]{other}A space here :)

\end{document}


Greetings.

• You are ending up with content that is just \scantokens{<a load of tokens>}, so you are not setting anything at point-of-use. You want \tl_rescan:nn. – Joseph Wright Jun 25 '19 at 7:10
• Also you don't actually seem to have any set-up – Joseph Wright Jun 25 '19 at 7:16
• I think I've misunderstood the operation of the rescan and rescan. Replacing \scantokens with \tl_rescan:nn {} {\l_tmpa_tl } does not get the same result. – Pablo González L Jun 25 '19 at 14:42
• @JosephWright: Can you explain to me a little bit how \tl_rescan works. If I can't use it in my case, your explanation would help me learn and close the question, greetings. – Pablo González L Jun 26 '19 at 18:26

January 1, 2020: Hello Pablo, happy new year to You and to everybody else! Stay safe and healthy!

The subject of the question is:

How do use {⟨setup⟩} in \tl_set_rescan:Nnn to replace \scantokens?

\tl_set_rescan:Nnn is based on \scantokens/ \tex_scantokens:D but the purposes of the two differ:

\tl_set_rescan:Nnn ⟨tl var⟩ {⟨setup⟩} {⟨tokens⟩} is intended to deliver a sequence of tokens that all got re-tokenized (but not processed/digested further) under a specific category-code-régime as the value/content of a token-list-variable.

\tl_rescan:nn {⟨setup⟩} {⟨tokens⟩} is intended to deliver a sequence of tokens that all got re-tokenized (but not processed/digested further) under a specific category-code-régime to the token-queue in the gullet of TeX for further digestion.

\scantokens{⟨tokens⟩} is intended to initialize re-digestion of a piece of .tex-input under the current category-code-régime.
Tokenization is only one aspect/stage of TeX's digestion of .tex-input.

With both commands the source of .tex-input for re-tokenization/re-digestion is the result of emulating the unexpanded writing of ⟨tokens⟩ to some emulated empty external text file.

\tl_set_rescan:Nnn ⟨tl var⟩ {⟨setup⟩} {⟨tokens⟩} does the following:

• Gather the three arguments (under current category-code-régime).
• Change the current category-code-régime according to ⟨setup⟩. This will affect how subsequent .tex-input is tokenized. This may affect the unexpanded-writing of control sequence tokens to external text files. (Changing category codes may lead to a one-letter control-sequence not being considered a control-word-token any more but being considered a control-symbol-token and vice versa. When unexpanded-writing control-word-tokens TeX does append a space-character. When unexpanded-writing control-symbol-tokens TeX does not append a space-character.)
• Set \newlinechar=\endlinechar.
• Emulate unexpanded-writing of the result of \detokenize{⟨tokens⟩} to some emulated external text-file.
• Read and tokenize (hereby obeying the changed category-code-régime) the entire emulated external text-file as follows:
TeX does not stop looking at the emulated external text-file and producing tokens and accumulating them (instead of digesting them) until it has seen/processed all characters contained in the emulated external text-file. When TeX has seen/processed all the characters contained in the emulated external text-file, then the accumulated tokens are stored in ⟨tl var⟩.
Be aware that tokens coming into being earlier in this process do not affect how subsequent characters of the emulated external text-file get tokenized. Tokens coming into being earlier in this process cannot affect how subsequent characters of the emulated external text-file get tokenized because tokens coming into being earlier are not digested but accumulated by TeX.
• Undo the change of the current category-code-régime and undo the change of \newlinechar.

\tl_rescan:nn {⟨setup⟩} {⟨tokens⟩} is almost like \tl_set_rescan:Nnn. The only difference is that the accumulated tokens are not stored in a ⟨tl var⟩. When everything is accumulated and the changes to the category code régime are undone, then all tokens at once are inserted into the token-queue in TeX's gullet (where expansion takes place) for further digestion.

\scantokens⟨general text⟩ / \tex_scantokens:D⟨general text⟩
⟨general text⟩⟨filler⟩{⟨balanced text⟩⟨right brace⟩ does the following:

• Emulate unexpanded-writing the tokens of ⟨balanced text⟩ to some emulated external text-file.
• Process that emulated external text-file as if it were an external text-file that is loaded via the \input-primitive.
Processing the emulated external text-file is initiated with the category-code-régime current at the time of starting to carry out \scantokens/\tex_scantokens:D.
Tokens hereby coming into being are not accumulated for later storage. Each token is pushed through TeX's digestion-tract immediately. Thus each token will be digested by TeX as usual. The result of digesting a token may be a change of the category code régime which in turn may affect how subsequent characters from the emulated external-text-file get tokenized.

Now you might have the impression that

\tl_rescan:nn {⟨setup⟩} {⟨tokens⟩}

is something like

⟨setup⟩\scantokens{⟨tokens⟩}

Both with \tl_rescan:nn and with \scantokens unexpanded writing of ⟨tokens⟩ to external text file is emulated.

The subtle difference is in how that emulated external text file is read back and tokenized:

With \tl_rescan:nn all lines/characters of the emulated external text file are tokenized at once. The resulting set of tokens is inserted into the token-queue in TeX's gullet "in one go".

With \scantokens digestion of a .tex-input-file takes place as usual:
Reading and tokenizing characters from the emulated external text file takes place only when the mouth needs characters for producing tokens.
The mouth produces tokens only when the gullet (the expansion-machinery) needs tokens.
The gullet sends tokens towards the stomach when the stomach needs tokens, e.g., for producing boxes, for performing whatsoever assignment.

Thus with

\scantokens\expandafter{\string\verb+text+}


the token \verb is digested. \verb causes tokenizing and "looking" at the next token (which is expected to be a character token), in this scenario the character-token +12, and (via some \lowercase-trickery) making the corresponding character (which in this scenario is +) active and reading/tokenizing/processing subsequent characters under verbatim-catcode-régime until having tokenized an active-+ denoting the end of the argument of \verb.

But with

\exp_args:Nno \tl_rescan:nn {} {\token_to_str:N \verb+text+}


the token \verb is not digested immediately. Instead it is accumulated like everything else.
In the end the token-sequence

\verbcontrol word token+12t11e11x11t11+12

is accumulated and inserted into the token-queue in TeX's gullet.

The problem is: \verb in this scenario switches the category code of + to active and keeps TeX tokenizing things under verbatim-catcode-régime until having tokenized an active-+. But TeX will not tokenize an active-+ as the second + in the sequence already got tokenized with category code 12(other). The end of \verb's argument cannot be found.

If I got it right, in your code the macro \_scontents_macro_to_seq: is intended to replace all endline-characters within the token-list-variable \l_tmpa_tl by newline-characters and to append the sequence \scantokens{total expansion of the content of the token-list-variable \l_tmpa_tl} to the variable \l_scontents_name_seq_tl.

Since the order in time in which, with respect to reading back/digesting the emulated external text file, the various digest operations take place differs for \scantokens/\tex_scantokens:D and \tl_rescan:nn/\tl_set_rescan:Nnn, it is not possible to emulate \scantokens{⟨tokens⟩} using \tl_rescan:nn or \tl_set_rescan:Nnn.

• First of all Very happy 2021, secondly, thanks a million for the explanation, the truth is that this was part of the questions (which were many) in the development of the package scontents and much of the code was adapted from ideas of the answers that you have placed in the forum using \scantokens, hopefully in some future expl3 will provide a function that is really equivalent to \scantokens. – Pablo González L Jan 1 at 23:45
• @PabloGonzálezL \scantokens is \tex_scantokens:D in expl3. I don't know why it doesn't have a better name there. ;-) – Ulrich Diez Jan 2 at 0:31
• Hehe... The :D means "Do not use" and strongly discouraged in package code :( – Pablo González L Jan 2 at 0:38
• @PabloGonzálezL Do not use/strongly discouraged = Using this may turn out very interesting - Using this may not just take you to LaTeX-heaven but may take you everywhere. ;-) – Ulrich Diez Jan 2 at 0:50
• All primitives have a :D name; it's a tricky primitive to use at the best of times, and the wrappers are meant to provide a reasonably-useful interface – Joseph Wright Jan 2 at 11:25