Inside \ExplSyntaxOn, ~ tilde char is set to have category code of 10 (space). But then \tl_set_rescan:Nnn gobbles every space input as ~.

  • If category code of space char is restored to 10 and then use to input space, then it works fine.
  • If category code of other char, for example dot char . is changed to 10 and then use this char to input space, the same problem appears.



% case 1, use ~, every space is gobbled
\tl_set_rescan:Nnn \x {} { ~ a ~ b ~ }
|\x| % print "|ab|"

% case 2, restore catcode of space char
\char_set_catcode_space:n {`\ }
\tl_set_rescan:Nnn \x {} { a b }
|\x| % print "| a b |"

% case 3, set catcode of . to 10
\char_set_catcode_space:n {`\.}
\tl_set_rescan:Nnn \x {} { . a . b . }
|\x| % print "|ab|"


That's expected.

As soon as a space is tokenized, for instance when a token list is formed for determining the argument to a command, space tokens are normalized with character code 32 and category code 10.

How does “rescanning” work? You have to think as if a virtual file is created and input, which TeX tokenizes again with the current category code array in force (the second argument to \tl_set_rescan:Nnn can contain code for changing the category codes). So a ~ in the input is read in as a space, which under \ExplSyntaxOn is 9 (ignored).

If you want to preserve spaces upon rescanning you can do

\tl_set_rescan:Nnn \l_tmpa_tl { \char_set_catcode_space:n {`\ } } { ~ a ~ b ~ }
\tl_analysis_show:N \l_tmpa_tl

that will output on the terminal

The token list \l_tmpa_tl contains the tokens:
>    (blank space  )
>  a (the letter a)
>    (blank space  )
>  b (the letter b)
>    (blank space  ).
  • So in my original example, ~ is first normalized with character code 32 and category code 10, resulting a normal space char in the virtual file. Then, while rescanning, this normal space is ignored since \ExplSyntaxOn sets \char_set_catcode_ignore:n { 32 } % space. Am I right? Jan 18 '20 at 13:44
  • @JohannesZ Yes, I think so. AFAIK, the “virtual file” comes from the \tl_to_str:n (aka \detokenize) mentioned in \tl_set_rescan:Nnn's documentation.
    – frougon
    Jan 18 '20 at 13:47
  • @frougon No, the virtual file comes from the \scantokens primitive.
    – egreg
    Jan 18 '20 at 13:51
  • @egreg Hmm, then is it correct that \tl_set_rescan:Nnn's documentation mentions \tl_to_str:n?
    – frougon
    Jan 18 '20 at 13:54
  • @frougon The argument is first stringified because otherwise control sequences would remain unmodified with \scantokens.
    – egreg
    Jan 18 '20 at 13:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.