1

I looked at interface3.pdf for learning about expl31.

Assume one uses the scratch-token-list-variable \l_tmpa_tl.

How can you pass the content of this variable as argument to another "function"2?

For example one would like to have the content of this variable processed by \scantokens{..}/\tex_scantokens:D or passed as <tokens> to \tl_rescan:nn {<setup>} {<tokens>} (although from the inscrutable explanation in interface3.pdf I didn't grasp the difference between \tex_scantokens:D and \tl_rescan:nn - seems \tl_rescan:nn does weird things about endline-chars or newline-chars or whatever).

How can this be done?

If all else fails: How many \expandafter do you need with \tl_use:N <tl var> to get the content of <tl var>?


1The main thing I've learned so far is that you often run into difficulties when you try to put the individual things you've read together into something that works. Something is always missing and then the fiddling starts, and you have to know TeX so well that you can implement the missing things yourself using TeX primitives. This in turn requires a level of TeX knowledge that makes the fact that expl3 is supposed to save you from having to go into depth obsolete.

2Why the heck do you call macros "functions" in expl3?

0
4

One should never use \scantokens in expl3 code.

One should never use \...:D control sequences in expl3 code.

What's the difference between \scantokens/\tex_scantokens:D and \tl_rescan:nn? It's hard to decide where to start, actually.

However, \tl_rescan:nn has two standard arguments, whereas \scantokens should be followed by a <general text> and already this is quite a big difference.

What's the purpose of the first argument? One can use it to make further category code settings to be applied when rescanning the second argument.

But the main difference is that \tl_rescan:nn does a good deal of work in order to fix the shortcomings of \scantokens (which has several).

Now the main point of your question. The expl3 language has a very precise way to do the job you want it to.

If you have a function \marsupialwallaby_foo:n that takes a standard braced argument, you can pass it the contents of a tl variable by doing

\cs_generate_variant:Nn \marsupialwallaby_foo:n { V }

and call the function as

\marsupialwallaby_foo:V \l_tmpa_tl

If you had previously said \tl_set:Nn \l_tmpa_tl { baz } the previous call would be equivalent to

\marsupialwallaby_foo:n { baz }

Of course, the current contents of the variable would be used.

In your case

\cs_generate_variant:Nn \tl_rescan:nn { nV }

will allow you to do

\tl_rescan:nV { } \l_tmpa_tl

Consult Part V, sections 1 to 3 of the interface3 manual.

By the way, the same result could be obtained by saying

\exp_args:NnV \tl_rescan:nn { } \l_tmpa_tl

but generating the variant is handier and recommended.

If you want to rescan the contents of \l_tmpa_tl and keep the result in the same variable, you can do

\cs_generate_variant:Nn \tl_set_rescan:Nnn { nnV }

\tl_set_rescan:NnV \l_tmpa_tl { } \l_tmpa_tl

It goes without saying that the \cs_generate_variant:Nn action needs to be performed only once at the beginning of the code that needs the variant.


Why do we use the term “function”? Because “macro” is too generic and doesn't really reflect the programming paradigm of expl3 that has “functions” and “variables”. Whether either of them are implemented as macros, primitives or registers it should be of no importance to the programmer.

2

If you read interface3.pdf carefully, you will stumble over the argument type V:

interface3.pdf, part V, section "3 Introducing the variants" explains the V-type:

3 Introducing the variants
The V type returns the value of a register, which can be one of tl, clist, int, skip, dim, muskip, or built-in TeX registers. The v type is the same except it first creates a control sequence out of its argument before returning the value.
In general, the programmer should not need to be concerned with expansion control. When simply using the content of a variable, functions with a V specifier should be used.
For those referred to by (cs)name, the v specifier is available for the same purpose. Only when specific expansion steps are needed, such as when using delimited arguments, should the lower-level functions with o specifiers be employed.

expl3.pdf, section "4 Expansion control" says:

V Value of a variable.
This means that the contents of the register in question is used as the argument, be it an integer, a length-type register, a token list variable or similar. The value is passed to the function as a braced token list. Can be applied to variables which have a \<var>_use:N function (other than floating points and boxes), and which therefore deliver a single “value”.

A possible answer to your question is:

Have expl3 generate a variant \tl_rescan:nV from \tl_rescan:nn and use the variant:

\cs_generate_variant:Nn \tl_rescan:nn {nV}
...
\tl_rescan:nV {<setup>} {\l_tmpa_tl}

About the difference between \tl_rescan:nn and \scantokens/\tex_scantokens:D:

In Knuth's analogy to a digestive process TeX has

  • eyes,
  • a digestive tract with a mouth, a gullet, a stomach and intestines.
  • the ability to produce tokens and to process them by means of its digestive organs.

TeX's eyes read the .tex-input-file linewise and place the characters of a line into the mouth. (The eyes hereby pre-process the sequence of characters coming from a line and this is important. But this is not important for understanding the crucial difference between \tl_rescan:nn and \scantokens/\tex_scantokens:D.) The mouth takes the characters for a set of instructions for producing tokens and sending them to the gullet. The tokens sent to the gullet form a "token-stream" whose elements are processed in TeX's gullet and/or in subsequent stations of TeX's digestive tract. Tokens can be control-sequence-tokens in different flavors or character-tokens in different flavors.

TeX's mouth produces tokens according to rules provided by characters' category-codes, by values of parameters like \endlinechar and by things that are "hardcoded" into the TeX-program (or LaTeX-program, if you use the variant where the set of macros that forms the LaTeX-format is loaded automatically).

Expansion of a(n expandable) token—i.e., replacement of that token (and probably those tokens that form its arguments) by other tokens—takes place while tokens are transported through TeX's gullet. (Unless expansion is suppressed, i.e., the gullet is "told" not to expand tokens.) TeX's gullet is the "expansion-station".

Assignments (defining macros, assigning values to \count-registers and the like) in that analogy take place in TeX's stomach. Producing a series of pages is also done by TeX's stomach.

The intestines transform each page into the form required by the output-file (.dvi-file/.pdf-file).

The final result of TeX's digestion-process is formed by the output-files (.pdf-file/.dvi-file, .log-file, auxiliary text files like the .aux-files and .toc-file and .lot/.lof-file etc) and the things that are written to the console.

Normally, the digestive stations work on demand and always deliver only just as much as is demanded:

When the mouth needs and thus requests characters, it requires them from the eyes and the eyes deliver the characters of a line of .tex-input. When the gullet needs and thus requests tokens which it can probably expand, it requires them from the mouth and the mouth delivers some.
When the stomach requires tokens, it requires them from the gullet and the gullet delivers some, hereby doing the great service of expansion if expansion is not suppressed.

\scantokens/\tex_scantokens:D works as follows:

It pretends writing without expansion the tokens that form its argument to an external text file and then causes TeX to focus its eyes on that "fake text file" as source of .tex-input instead of focusing them on the actual .tex-input file. I call this a "fake text file" because the data is not stored on a device for fixed data storage but is held in the computer's RAM.

The digestive processes of TeX are carried out normally, except that the eyes are not focused on the actual .tex-input file as source of .tex-input but are focused on what is delivered by \scantokens'/\tex_scantokens:D's pretended unexpanded writing of its argument to the piece of RAM holding the data of the fake-text-file.

As with any other source of .tex-input with \scantokens/\tex_scantokens:D's fake-text-file as source of .tex-input characters coming from lines of .tex-input are placed into the mouth on demand only and the mouth produces tokens for the gullet on demand only and the gullet delivers tokens to the stomach on demand only.

Thus with \scantokens/\tex_scantokens:D as source of .tex-input time-intervals of on-demand-placing characters into TeX's mouth are followed by time-intervals in which the characters in TeX's mouth are digested which leads to producing tokens and executing/carrying out the things denoted by these tokens.
This may, e.g., lead to carrying out directives for changing how to henceforth produce tokens from the characters that get placed into TeX's mouth. "Directives for changing how to henceforth produce tokens may be: Changing category codes, changing values of parameters like \endlinechar, etc. Commands like \verb and environments like verbatim trigger temporarily changing category codes. Such changes also affect how subsequent things of the fake-text-file produced from \scantokens'/\tex_scantokens:D's argument get tokenized.

If I got it right \tl_rescan:nn like \scantokens/\tex_scantokens:D triggers the creation of a fake text file and also triggers TeX's eyes to focus on that fake text file.

Unlike with \scantokens/\tex_scantokens:D where tokens are produced from the characters of the fake text file on demand only, with \tl_rescan:nn there is no mixing of time intervals of putting characters of .tex input into the mouth of TeX with time intervals of digesting these characters / with time intervals of creating, expanding and further processing of tokens.
With \tl_rescan:nn tokens are produced from all characters/lines coming from the fake text file in one go. All these tokens then are appended to the gullet's token stream and are ready for further digestion.

A consequence of this is:

Directives within \tl_rescan:nn's ⟨tokens⟩-argument for (temporarily) changing category codes (e.g., things like \verb or the verbatim-environment bring along such directives) do not affect how subsequent things of \tl_rescan:nn's ⟨tokens⟩-argument get re-tokenized because these subsequent things are already re-tokenized when the tokens forming the directives reach TeX's stomach and get carried out.


The main thing I've learned when reading computer manuals like the TeXBook or interface3.pdf is:

You need to read them like a lawyer weighing the meaning of each syllable as he examines a contract submitted to him for review for possible pitfalls. ;-)

Unlike a lawyer, however, you don't have to do this in order to detect possible malice, but you have to do it to not miss anything important/to not overlook important implications of what is worded. ;-)

8
  • I beg to disagree. The TeXBook is carefully written. Maybe you have to think a bit like a mathematician, but definitely not like a lawyer. – user228539 Nov 17 '20 at 2:13
  • @abcdefg The TeXBook indeed is, among other things, a very carefully written compilation of verbal utterances, in which no syllable, word or subordinate clause may be ignored, if all the implications inherent in the utterances are to be correctly understood. Properly capturing what a verbal utterance implies is one of the main tasks of lawyers who review contracts for the resulting possibilities of handling things, pitfalls and loopholes. – Ulrich Diez Nov 18 '20 at 0:07
  • I beg to disagree, again. Many lawyers try to avoid clarity in order to defend their quasi-monopoly. The TeX book, on the other hand, has helped to build up an impressively large (La)TeX user base, and helps the users to acquire full control over the "beast". Notice that this refers to the TeX book, not to answers which start with "One should never .... " without any explanation. Notice that I am not saying that you were making such statements. – user228539 Nov 18 '20 at 0:13
  • 1
    @abcdefg MY utterance is not so much about what texts one reads but about how to go at reading them. – Ulrich Diez Nov 18 '20 at 1:07
  • 2
    @abcdefg Knuth in the TeXbook explicitely states that in the early chapters he is not always saying the truth - Preface of the TeXbook says: "Another noteworthy characteristic of this manual is that it doesn’t always tell the truth. When certain concepts of TeX are introduced informally, general rules will be stated; afterwards you will find that the rules aren’t strictly true. In general, the later chapters contain more reliable information than the earlier ones do. The author feels that this technique of deliberate lying will actually make it easier for you to learn the ideas." – Ulrich Diez Nov 18 '20 at 1:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.