Space counting in expl3 is strange

Question

The function \str_count_spaces:n{} should count the number of spaces in its parameter.

For me it outputted unexpected results.

\str_count_spaces:n{he llo} %->0

However these were okay,

\str_count_spaces:n{he\ llo} %->1
\str_count_spaces:n{he~llo} %->1
\str_count_spaces:n{he\char{32}llo} %->1

Did I misunderstand the definition of this function?

Answer 1

Yes, you misunderstood the concept of strings in expl3.

A TeX string (and thus an expl3 string) is a series of characters which have category code 12 (“other”) with the exception of space characters which have category code 10 (“space”). Thus at a technical level, a TeX string is a token list with the appropriate category codes. In this documentation, these are simply referred to as strings.

There's a problem: to understand what this means you need to understand what "token" in TeX means, which ultimately boils down to reading TeXbook etc.... (resources)

Anyway, so \char or \ are not character tokens, but all the str function converts its arguments to string. You can view what the string will be with \str_show:n:

\documentclass{article}
\begin{document}

\ExplSyntaxOn
\str_show:n{he\ llo}
\str_show:n{he~llo}
\str_show:n{he\char{32}llo}
\ExplSyntaxOff

\end{document}

Result:

he\ llo
he llo
he\char {32}llo

Because of how \detokenize primitive works in eTeX (the behavior is inherited from whatever function used to implement converting token list to string in TeX source code, in expl3 documentation it's mentioned that the conversion is done with \tl_to_str:n where the description of \detokenize is copied there) a space is added after \char, which has nothing to do with the fact that the number is 32 and it typesets a space when run normally.

Answer 2

Within \ExplSyntaxOn and \ExplSyntaxOff all spaces are ignored.

To insert an explicit space use ~ (which as you note works).

You can create a LaTeX command that wraps the expl3 macro to count spaces in a non-strange way outside of of the Expl3 programming environment:

\documentclass{article}
\ExplSyntaxOn
\NewDocumentCommand \CountSpaces { m }
  {
    \str_count_spaces:n { #1 }
  }
\ExplSyntaxOff
\begin{document}
\CountSpaces{he llo} % -> 1
\end{document}