# Inside \ExplSyntaxOn/Off pair, \tl_set_rescan:Nnn gobbles every space input as ~

Inside \ExplSyntaxOn, ~ tilde char is set to have category code of 10 (space). But then \tl_set_rescan:Nnn gobbles every space input as ~.

• If category code of space char is restored to 10 and then use to input space, then it works fine.
• If category code of other char, for example dot char . is changed to 10 and then use this char to input space, the same problem appears.

Example:

\documentclass{article}
\usepackage{expl3}

\begin{document}
\ExplSyntaxOn
% case 1, use ~, every space is gobbled
\tl_set_rescan:Nnn \x {} { ~ a ~ b ~ }
|\x| % print "|ab|"

% case 2, restore catcode of space char
\group_begin:
\char_set_catcode_space:n {\ }
\tl_set_rescan:Nnn \x {} { a b }
|\x| % print "| a b |"
\group_end:

% case 3, set catcode of . to 10
\group_begin:
\char_set_catcode_space:n {\.}
\tl_set_rescan:Nnn \x {} { . a . b . }
|\x| % print "|ab|"
\group_end:

\ExplSyntaxOff
\end{document}


That's expected.

As soon as a space is tokenized, for instance when a token list is formed for determining the argument to a command, space tokens are normalized with character code 32 and category code 10.

How does “rescanning” work? You have to think as if a virtual file is created and input, which TeX tokenizes again with the current category code array in force (the second argument to \tl_set_rescan:Nnn can contain code for changing the category codes). So a ~ in the input is read in as a space, which under \ExplSyntaxOn is 9 (ignored).

If you want to preserve spaces upon rescanning you can do

\tl_set_rescan:Nnn \l_tmpa_tl { \char_set_catcode_space:n {\ } } { ~ a ~ b ~ }
\tl_analysis_show:N \l_tmpa_tl


that will output on the terminal

The token list \l_tmpa_tl contains the tokens:
>    (blank space  )
>  a (the letter a)
>    (blank space  )
>  b (the letter b)
>    (blank space  ).

• So in my original example, ~ is first normalized with character code 32 and category code 10, resulting a normal space char in the virtual file. Then, while rescanning, this normal space is ignored since \ExplSyntaxOn sets \char_set_catcode_ignore:n { 32 } % space. Am I right? Jan 18 '20 at 13:44
• @JohannesZ Yes, I think so. AFAIK, the “virtual file” comes from the \tl_to_str:n (aka \detokenize) mentioned in \tl_set_rescan:Nnn's documentation. Jan 18 '20 at 13:47
• @frougon No, the virtual file comes from the \scantokens primitive. Jan 18 '20 at 13:51
• @egreg Hmm, then is it correct that \tl_set_rescan:Nnn's documentation mentions \tl_to_str:n? Jan 18 '20 at 13:54
• @frougon The argument is first stringified because otherwise control sequences would remain unmodified with \scantokens`. Jan 18 '20 at 13:56