In my opinion, l3 strings are special token lists whose where all characters has category code 12 (except space), and most interfaces in l3tl and l3str are similar. So, if I just want to process "plain" text, what exactly is their difference?
For example, should I use token list or string for option names, cross-reference labels? What is the best practice?
l3 strings are for low level tex programming constructs not for natural language texts, note in particular that in pdftex you can not have accented (or any non-ascii) characters in a string as they require active characters and inputenc processing.
l3kernel to token lists in expl3? Defining one as NewDocumentCommand\tl_foo{}{token list}, that is with no argument, is cumbersome.
inputenc restriction you mentioned still hold? Or can now L3 strings handle properly non-ascii characters?
inputenc and l3 str is a tex string as returned by \string or \detokenize so all tokens are catcode 12 or 10. That means it can hold a utf8 encoded Unicode string and be used in say a filename or pdf internal string, but it won't typeset as unicode characters unless you re-parse it by writing it out and reading it back or using \scantokens or some such. this is basically not a latex3 restriction it's an engine one. \string\ééé may look like \ééé if you \typeout but will typeset as 7 catcode12 bytes.
Commented
Jul 13, 2021 at 19:28
The interface documentation itself is not entirely clear about that.
Part VII The l3str package: Strings, p. 49
TeX associates each character with a category code: as such, there is no concept of a “string” as commonly understood in many other programming languages. However, there are places where we wish to manipulate token lists while in some sense “ignoring” category codes: this is done by treating token lists as strings in a TeX sense.
A TeX string (and thus an
expl3string) is a series of characters which have category code 12 (“other”) with the exception of space characters which have category code 10 (“space”). Thus at a technical level, a TeX string is a token list with the appropriate category codes. In this documentation, these are simply referred to as strings.String variables are simply specialised token lists, but by convention should be named with the suffix
...str. Such variables should contain characters with category code 12 (other), except spaces, which have category code 10 (blank space). All the functions in this module which accept a token list argument first convert it to a string using\tl_to_str:nfor internal processing, and do not treat a token list or the corresponding string representation differently.As string is a subset of the more general token list, it is sometimes unclear when one should be used over the other. Use a string variable for data that isn’t primarily intended for typesetting and for which a level of protection from unwanted expansion is suitable. This data type simplifies comparison of variables since there are no concerns about expansion of their contents.
