How can I run scantokens with modified environment, but use the normal environment within the scanned code?

Question

For example if I run

%! TEX program = lualatex
\documentclass{article}
\begin{document}


{% prepare a token list <a> (assume this is provided by user)
\catcode 13=12\relax  % 12:other
\global\def\a{line 1^^M^^Mline 2^^M^^Mnewlinechar=\the\newlinechar^^M}%
}

% scan \a, but with newlinechar=13...
\newlinechar=13\relax
\expandafter\scantokens\expandafter{\a}
% but then the code inside should still see newlinechar=10.

\end{document}

each part separated by the character with charcode 13 will be written as a separate line; however the code inside will see \newlinechar=13.

Assume I want the code inside to see the value to be equal to the normal value i.e. 10 instead. How can I achieve that? (the command doesn't seem to provide any obvious way to do it.)

(Also assume that the code inside is an arbitrary user-provided content that should probably not be touched, and it may contain catcode-changing command that should be obeyed.)

Answer 1

Alright, there are a few ways I can figure out.

Modify the input to suit the newlinechar

As suggested by Skillmon.

\cs_generate_variant:Nn \tl_replace_all:Nnn { Nee }
\tl_replace_all:Nee \a { \char_generate:nn { 13 } { 12 } } { \char_generate:nn { \newlinechar } { 12 } }
\tl_set_rescan:Nno \a {} { \a }

Issue: get into problems if \newlinechar is -1 or some other weird value e.g. /, and the user code happen to have a / as well.

Expand \scantokens once then revert newlinechar

Looks like the o-expansion of \scantokens is similar to the o-expansion of \@@input, put the content in front of the input stream.

This will work for the example in the question. (before executing \endgroup expand the \scantokens once. The other one is to expand the \a. Side note, \scantokens is a primitive that expand the following tokens until it sees a {, so another \expandafter before it isn't necessary.)

%! TEX program = lualatex
\documentclass{article}
\begin{document}


{% prepare a token list <a> (assume this is provided by user)
\catcode 13=12\relax  % 12:other
\global\def\a{line 1^^M^^Mline 2^^M^^Mnewlinechar=\the\newlinechar^^M}%
}

% scan \a, but with newlinechar=13...
\begingroup\newlinechar=13\relax\expandafter\endgroup
\scantokens\expandafter{\a}
% but then the code inside should still see newlinechar=10.

\end{document}

Use an active character to revert newlinechar inside the rescanned tokens

%! TEX program = lualatex
\documentclass{article}
\begin{document}


{% prepare a token list <a> (assume this is provided by user)
\catcode 13=12\relax  % 12:other
\global\def\a{line 1^^M^^Mline 2^^M^^Mnewlinechar=\the\newlinechar^^M}%
}

\begingroup
\catcode `\~\active
\let~\endgroup
\newlinechar=13\relax
\scantokens\expandafter{\expandafter~\a}

\end{document}

I think this method doesn't have any disadvantage (if the default catcode of ~ is not active it will be reset when the group is closed, and \endlinechar doesn't matter). Basically

• open a group
• make ~ active, and make it close the group when executed
• set \newlinechar
• pass ~⟨content⟩ to \scantokens. When the rescanned ~ is executed the group will be closed.