expl3 - how can you pass the content of a token-list-variable as argument to another "function"?

Question

I looked at interface3.pdf for learning about expl3¹.

Assume one uses the scratch-token-list-variable \l_tmpa_tl.

How can you pass the content of this variable as argument to another "function"²?

For example one would like to have the content of this variable processed by \scantokens{..}/\tex_scantokens:D or passed as <tokens> to \tl_rescan:nn {<setup>} {<tokens>} (although from the inscrutable explanation in interface3.pdf I didn't grasp the difference between \tex_scantokens:D and \tl_rescan:nn - seems \tl_rescan:nn does weird things about endline-chars or newline-chars or whatever).

How can this be done?

If all else fails: How many \expandafter do you need with \tl_use:N <tl var> to get the content of <tl var>?

---

¹The main thing I've learned so far is that you often run into difficulties when you try to put the individual things you've read together into something that works. Something is always missing and then the fiddling starts, and you have to know TeX so well that you can implement the missing things yourself using TeX primitives. This in turn requires a level of TeX knowledge that makes the fact that expl3 is supposed to save you from having to go into depth obsolete.

²Why the heck do you call macros "functions" in expl3?

Answer 1

One should never use \scantokens in expl3 code.

One should never use \...:D control sequences in expl3 code.

What's the difference between \scantokens/\tex_scantokens:D and \tl_rescan:nn? It's hard to decide where to start, actually.

However, \tl_rescan:nn has two standard arguments, whereas \scantokens should be followed by a <general text> and already this is quite a big difference.

What's the purpose of the first argument? One can use it to make further category code settings to be applied when rescanning the second argument.

But the main difference is that \tl_rescan:nn does a good deal of work in order to fix the shortcomings of \scantokens (which has several).

Now the main point of your question. The expl3 language has a very precise way to do the job you want it to.

If you have a function \marsupialwallaby_foo:n that takes a standard braced argument, you can pass it the contents of a tl variable by doing

\cs_generate_variant:Nn \marsupialwallaby_foo:n { V }

and call the function as

\marsupialwallaby_foo:V \l_tmpa_tl

If you had previously said \tl_set:Nn \l_tmpa_tl { baz } the previous call would be equivalent to

\marsupialwallaby_foo:n { baz }

Of course, the current contents of the variable would be used.

In your case

\cs_generate_variant:Nn \tl_rescan:nn { nV }

will allow you to do

\tl_rescan:nV { } \l_tmpa_tl

Consult Part V, sections 1 to 3 of the interface3 manual.

By the way, the same result could be obtained by saying

\exp_args:NnV \tl_rescan:nn { } \l_tmpa_tl

but generating the variant is handier and recommended.

If you want to rescan the contents of \l_tmpa_tl and keep the result in the same variable, you can do

\cs_generate_variant:Nn \tl_set_rescan:Nnn { nnV }

\tl_set_rescan:NnV \l_tmpa_tl { } \l_tmpa_tl

It goes without saying that the \cs_generate_variant:Nn action needs to be performed only once at the beginning of the code that needs the variant.

---

Why do we use the term “function”? Because “macro” is too generic and doesn't really reflect the programming paradigm of expl3 that has “functions” and “variables”. Whether either of them are implemented as macros, primitives or registers it should be of no importance to the programmer.

Answer 2

If you read interface3.pdf carefully, you will stumble over the argument type V:

interface3.pdf, Chapter 5, section "5.3 Introducing the variants" explains the V-type:

5.3 Introducing the variants
The V type returns the value of a register, which can be one of tl, clist, int, skip, dim, muskip, or built-in TeX registers. The v type is the same except it first creates a control sequence out of its argument before returning the value.
In general, the programmer should not need to be concerned with expansion control. When simply using the content of a variable, functions with a V specifier should be used.
For those referred to by (cs)name, the v specifier is available for the same purpose. Only when specific expansion steps are needed, such as when using delimited arguments, should the lower-level functions with o specifiers be employed.

expl3.pdf, section "4 Expansion control" says:

V Value of a variable.
This means that the contents of the register in question is used as the argument, be it an integer, a length-type register, a token list variable or similar. The value is passed to the function as a braced token list. Can be applied to variables which have a \<var>_use:N function (other than floating points and boxes), and which therefore deliver a single “value”.

A possible answer to your question is:

Have expl3 generate a variant \tl_rescan:nV from \tl_rescan:nn and use the variant:

\cs_generate_variant:Nn \tl_rescan:nn {nV}
...
\tl_rescan:nV {<setup>} {\l_tmpa_tl}

---

About the difference between \tl_rescan:nn and \scantokens/\tex_scantokens:D:

In Knuth's analogy to a digestive process TeX has

• eyes,
• a digestive tract with a mouth, a gullet, a stomach and intestines.
• the ability to produce tokens and to process them by means of its digestive organs.

TeX's eyes read the .tex-input-file linewise and place the characters of a line into the mouth. (The eyes hereby pre-process the sequence of characters coming from a line and this is important. But this is not important for understanding the crucial difference between \tl_rescan:nn and \scantokens/\tex_scantokens:D.) The mouth takes the characters for a set of instructions for producing tokens and sending them to the gullet. The tokens sent to the gullet form a "token-stream" whose elements are processed in TeX's gullet and/or in subsequent stations of TeX's digestive tract. Tokens can be control-sequence-tokens in different flavors or character-tokens in different flavors.

TeX's mouth produces tokens according to rules provided by characters' category-codes, by values of parameters like \endlinechar and by things that are "hardcoded" into the TeX-program (or LaTeX-program, if you use the variant where the set of macros that forms the LaTeX-format is loaded automatically).

Expansion of a(n expandable) token—i.e., replacement of that token (and probably those tokens that form its arguments) by other tokens—takes place while tokens are transported through TeX's gullet. (Unless expansion is suppressed, i.e., the gullet is "told" not to expand tokens.) TeX's gullet is the "expansion-station".

Assignments (defining macros, assigning values to \count-registers and the like) in that analogy take place in TeX's stomach. Producing a series of pages is also done by TeX's stomach.

The intestines transform each page into the form required by the output-file (.dvi-file/.pdf-file).

The final result of TeX's digestion-process is formed by the output-files (.pdf-file/.dvi-file, .log-file, auxiliary text files like the .aux-files and .toc-file and .lot/.lof-file etc) and the things that are written to the console.

Normally, the digestive stations work on demand and always deliver only just as much as is demanded:

When the mouth needs and thus requests characters, it requires them from the eyes and the eyes deliver the characters of a line of .tex-input. When the gullet needs and thus requests tokens which it can probably expand, it requires them from the mouth and the mouth delivers some.
When the stomach requires tokens, it requires them from the gullet and the gullet delivers some, hereby doing the great service of expansion if expansion is not suppressed.

\scantokens/\tex_scantokens:D works as follows:

It pretends writing without expansion the tokens that form its argument to an external text file and then causes TeX to focus its eyes on that "fake text file" as source of .tex-input instead of focusing them on the actual .tex-input file. I call this a "fake text file" because the data is not stored on a device for fixed data storage but is held in the computer's RAM.

The digestive processes of TeX are carried out normally, except that the eyes are not focused on the actual .tex-input file as source of .tex-input but are focused on what is delivered by \scantokens'/\tex_scantokens:D's pretended unexpanded writing of its argument to the piece of RAM holding the data of the fake-text-file.

As with any other source of .tex-input with \scantokens/\tex_scantokens:D's fake-text-file as source of .tex-input characters coming from lines of .tex-input are placed into the mouth on demand only and the mouth produces tokens for the gullet on demand only and the gullet delivers tokens to the stomach on demand only.

Thus with \scantokens/\tex_scantokens:D as source of .tex-input time-intervals of on-demand-placing characters into TeX's mouth are followed by time-intervals in which the characters in TeX's mouth are digested which leads to producing tokens and executing/carrying out the things denoted by these tokens.
This may, e.g., lead to carrying out directives for changing how to henceforth produce tokens from the characters that get placed into TeX's mouth. "Directives for changing how to henceforth produce tokens may be: Changing category codes, changing values of parameters like \endlinechar, etc. Commands like \verb and environments like verbatim trigger temporarily changing category codes. Such changes also affect how subsequent things of the fake-text-file produced from \scantokens'/\tex_scantokens:D's argument get tokenized.

If I got it right \tl_rescan:nn like \scantokens/\tex_scantokens:D triggers the creation of a fake text file and also triggers TeX's eyes to focus on that fake text file.

Unlike with \scantokens/\tex_scantokens:D where tokens are produced from the characters of the fake text file on demand only, with \tl_rescan:nn there is no mixing of time intervals of putting characters of .tex input into the mouth of TeX with time intervals of digesting these characters / with time intervals of creating, expanding and further processing of tokens.
With \tl_rescan:nn tokens are produced from all characters/lines coming from the fake text file in one go. All these tokens then are appended to the gullet's token stream and are ready for further digestion.

A consequence of this is:

Directives within \tl_rescan:nn's ⟨tokens⟩-argument for (temporarily) changing category codes (e.g., things like \verb or the verbatim-environment bring along such directives) do not affect how subsequent things of \tl_rescan:nn's ⟨tokens⟩-argument get re-tokenized because these subsequent things are already re-tokenized when the tokens forming the directives reach TeX's stomach and get carried out.

---

To the question: "Why the heck do you call macros "functions" in expl3?"

The term "function" is not just an expl3-jargon-synonym for "macro".

Strictly spoken a macro token is a token which is gone and replaced by the tokens forming its ⟨replacement text⟩ (with arguments replacing the ⟨replacement text⟩'s parameters #1, #2, ...) after triggering one step of expansion.

A "function" is something that is determined to yield a result (usually in terms of a set of tokens) at some stage of processing/expanding.

A "function" can, e.g., be a tricky implementation in TeX of some algorithm, whereby the implementation might as well consist of definitions of several (internal) macros.
Carrying out a function, which might probably be an expansion-driven tail-recursive loop or the like, may mean triggering some step(s) of expansion so that the result is not necessarily delivered after exactly one expansion-step but delivering the result may require triggering several expansion-steps.

---

The main thing I've learned when reading computer manuals like the TeXBook or interface3.pdf is:

You need to read them like a lawyer weighing the meaning of each syllable as he examines a contract submitted to him for review for possible pitfalls. ;-)

Unlike a lawyer, however, you don't have to do this in order to detect possible malice, but you have to do it to not miss anything important/to not overlook important implications of what is worded. ;-)