4

In my opinion, l3 strings are special token lists whose where all characters has category code 12 (except space), and most interfaces in l3tl and l3str are similar. So, if I just want to process "plain" text, what exactly is their difference?

For example, should I use token list or string for option names, cross-reference labels? What is the best practice?

1
  • 1
    What do you mean by plain text? Verbatim material or text in the document? For option names I would use token lists (because I want my users to use macros too), same for labels.
    – TeXnician
    Aug 15, 2018 at 11:53

2 Answers 2

4

l3 strings are for low level tex programming constructs not for natural language texts, note in particular that in pdftex you can not have accented (or any non-ascii) characters in a string as they require active characters and inputenc processing.

5
  • 3
    Perhaps add a simple 'Do X or Y' statement: I'd go with 'Strings are for data that will never be typeset, for example file names, identifiers, etc.: if the material may be used in typesetting, it should be a token list'.
    – Joseph Wright
    Aug 17, 2018 at 7:36
  • Is there an equivalent in l3kernel to token lists in expl3? Defining one as NewDocumentCommand\tl_foo{}{token list}, that is with no argument, is cumbersome.
    – Erwann
    Oct 4, 2019 at 21:31
  • David, may I ask one thing about your answer? Now that UTF-8 is the default input encoding for pdfTeX, does the inputenc restriction you mentioned still hold? Or can now L3 strings handle properly non-ascii characters?
    – gusbrs
    Jul 13, 2021 at 19:12
  • 1
    @gusbrs utf-8 default doesn't change anything, it's the same code as used with inputenc and l3 str is a tex string as returned by \string or \detokenize so all tokens are catcode 12 or 10. That means it can hold a utf8 encoded Unicode string and be used in say a filename or pdf internal string, but it won't typeset as unicode characters unless you re-parse it by writing it out and reading it back or using \scantokens or some such. this is basically not a latex3 restriction it's an engine one. \string\ééé may look like \ééé if you \typeout but will typeset as 7 catcode12 bytes. Jul 13, 2021 at 19:28
  • @DavidCarlisle I see. Thanks for answering. Let me be explicit for what I'm considering. I'd like to store, pass around and compare reference labels and escape the usual trouble with babel active characters. Would it be safe (and flexible) to store those as L3 strings? They are not meant for typesetting, but would any other restriction upon them crawl back in somewhere else?
    – gusbrs
    Jul 13, 2021 at 20:16
2

The interface documentation itself is not entirely clear about that.

Part VII The l3str package: Strings, p. 49

TeX associates each character with a category code: as such, there is no concept of a “string” as commonly understood in many other programming languages. However, there are places where we wish to manipulate token lists while in some sense “ignoring” category codes: this is done by treating token lists as strings in a TeX sense.

A TeX string (and thus an expl3 string) is a series of characters which have category code 12 (“other”) with the exception of space characters which have category code 10 (“space”). Thus at a technical level, a TeX string is a token list with the appropriate category codes. In this documentation, these are simply referred to as strings.

String variables are simply specialised token lists, but by convention should be named with the suffix ...str. Such variables should contain characters with category code 12 (other), except spaces, which have category code 10 (blank space). All the functions in this module which accept a token list argument first convert it to a string using \tl_to_str:n for internal processing, and do not treat a token list or the corresponding string representation differently.

As string is a subset of the more general token list, it is sometimes unclear when one should be used over the other. Use a string variable for data that isn’t primarily intended for typesetting and for which a level of protection from unwanted expansion is suitable. This data type simplifies comparison of variables since there are no concerns about expansion of their contents.


screenshot of the above text from the PDF manual with "it is sometimes unclear when one should be used over the other" highlighted

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.