LaTeX3: How to get reliable control over the level of expansion?

Question

Say you want to write LaTeX3 code that manages user data. Sometimes you want to process that data, but other times you just want to store it, move it around and return it unchanged (when developing a data-structure, for instance).

In the latter case you often need to expand down to the user-provided data exactly, but no further, since such data is not always meant for typesetting. Users should be able to store functions inside your data-structure and be sure that when returned, they'll still behave identically.

The problem is, many LaTeX3 functions give no guarantees about the number of expansions necessary to get a specific result.

For example, I had a token list variable \x containing three consecutive brace-groups with user-data. I wanted to put the content of the third group into a variable \y. This is the code I came up with:

\exp_args:NNNo \exp_args:NNo \tl_set:No \y {\exp_last_unbraced:No \use_iii:nnn \x}

I found out that \exp_last_unbraced:No requires 2 expansions and \use_iii:nnn requires one. But this will only work until the implementation of either of those functions silently changes to require a different number of expansions. Then the above code might use one too many or one too few.

How can I get more reliable control over this kind of expansion? I imagine it might have something to do with \exp_not:n and family, but it would be very helpful to have a guide to their proper use.

---

Edit: Rephrased Question

As observed by existing answers, the answer is of course not to rely on an exact number of expansions. So my question would be better phrased as:

Is there a generally recommended way of 'handling' data that makes it easier to distinguish between expansion until the original level and expansion beyond the original level?

Attempt at an answer

Even before asking the question, I imagined it might have something to do with packaging the data in \exp_not:n {-}. Then if you do an :x expansion, you get exactly the right data back. The problem is, :f expansion, :c expansion, :v expansion, etc. will happily go past this 'barrier' and eat into the data:

\ExplSyntaxOn
    \cs_generate_variant:Nn \tl_to_str:n {x}
    \cs_generate_variant:Nn \tl_to_str:n {f}

    \tl_new:N \l_external_tl
    \tl_new:N \l_data_tl
    \tl_new:N \l_internal_tl
    \tl_set:Nn \l_external_tl  {\l_data_tl}
    \tl_set:Nn \l_data_tl      {\exp_not:n{\l_internal_tl}}
    \tl_set:Nn \l_internal_tl  {too~far}

    \noindent\tt
    \tl_to_str:x {\l_external_tl}\\  %  \l_internal_tl  %  good
    \tl_to_str:f {\l_external_tl}\\  %  too far         %  bad
\ExplSyntaxOff

So my latest idea is a different kind of 'barrier'. Put the data in a token list variable with a unique csname. And then put only the csname in your internal structures. Then no power in the world can expand the data until a :c-related expansion is used. Even better, simply use a :v expansion to get exactly to the data and no further:

\ExplSyntaxOn
    \cs_generate_variant:Nn \tl_to_str:n {x}
    \cs_generate_variant:Nn \tl_to_str:n {f}
    \cs_generate_variant:Nn \tl_to_str:n {v}

    \int_zero_new:N \g__barrier_int
    \cs_new_protected:Nn \tl_set_barrier:Nn {
        \int_gincr:N \g__barrier_int
        \tl_set:cn {barrier(\int_use:N\g__barrier_int)_tl} {#2}
        \tl_set:Nx #1 {barrier(\int_use:N\g__barrier_int)_tl}
    }

    \tl_new:N \l_external_tl
    \tl_new:N \l_data_tl
    \tl_new:N \l_internal_tl
    \tl_set:Nn         \l_external_tl {\l_data_tl}
    \tl_set_barrier:Nn \l_data_tl     {\l_internal_tl}
    \tl_set:Nn         \l_internal_tl {too~far}

    \noindent\tt
    \tl_to_str:x {\l_external_tl}\\  %  barrier(1)      %  good
    \tl_to_str:f {\l_external_tl}\\  %  barrier(1)      %  good
    \tl_to_str:v {\l_external_tl}\\  %  \l_internal_tl  %  good
\ExplSyntaxOff

This way you can pile on fully-expandable operations to your hearts content and just do a :v expansion to get to the data. It's like pointer-redirection.

I will probably write a small package to better facilitate this and use it for my own LaTeX3 programs, unless someone can give me a better option. Let me know what you think.

Answer 1

As @egreg already said you are probably using the wrong data type as a starting point. The idea behind expl3 is to avoid using expansion based programming within your programs altogether (or more precisely limit it to very well-defined places).

so the "programming dicipline" would be something like

1. define how the external representation of the user data looks like that you are interested in (in your example 3 brace groups, say)
2. define an internal data structure for the user data that you want to manipulate
3. define a transformation process from external to internal
4. then use only the internal structure for manipulation and provide the necessary mutator functions for your data structure (unless you use an existing one that already has the tools like prop)

Now clearly there is some expansion necessary at some point, e.g., the user input might get scanned into some tl variable and you need to get it out from there to put it into your internal data structure but this step 3. should happen preferably once and could then use simple expansion rules, e.g.,

   \exp_after:wN \helvens_store_user_data:nnn \x 

or if the data come in via some document-level argument via

   \helvens_store_user_data:nnn #1   % with #1 being expected to be 3 brace groups

but neither the \x nor the #1 should be long-living and passed around in your code unless you declare that \xis a variable of your data structure type and then you should have some manipulation functions that go:

   \helvens_retrieve_third:NN \x\y

Yes, such a function would need to do some low-level expansion (due to the choice of your data structure) but those could be handled with the tools available (relying on the documented expansions that certain functions need), eg.

   \cs_new_protected:Npn \helvens_retrieve_third:NN #1 {
     \exp_after:wN \__helvens_retrieve_third:nnnN #1 
   }
   \cs_new:Npn \__helvens_retrieve_third:nnnN #1#2#3#4 {
      \tl_set:Nn #4{#3}
   }

and similar for any other functions that you want/need to manipulate your data structure.

But please don't go for something like

   \exp_args:NNNo \exp_args:NNo \tl_set:No \y {\exp_last_unbraced:No \use_iii:nnn \x}

in the middle of your programming code to avoid setting up data structure manipulation functions.

Having said all this ... you make some valid points here and we should

• better document this concept/approach
• document for those expansion functions that you reasonably need for setting up data structures how their expansion behavior is precisely; it is only a small set that should be used for this and most of not all of them are of type 1 expansion

In one of the comments it was mentioned that such a solution as given by egreg or myself would "perhaps" not work in some situations. If so I would suggest to provide those scenarios as additional questions. In our opinion this approach should be generally applicable.

Answer 2

I'd say that you're using the wrong data type, in this case. However, you don't need to control how many expansions you need to do:

\RequirePackage{expl3}
\ExplSyntaxOn
\tl_new:N \g_helvens_user_tl
\tl_gset:Nn \g_helvens_user_tl { { foo } { bar } { baz } }
\tl_new:N \l_helvens_third_tl

\cs_new_protected:Npn \helvens_get_third:NN #1 #2
 {
  \exp_last_unbraced:NNV \__helvens_get_third_aux:Nnnn #1 #2
 }

\cs_new_protected:Npn \__helvens_get_third_aux:Nnnn #1 #2 #3 #4
 {
  \tl_set:Nn #1 { #4 }
 }

\helvens_get_third:NN \l_helvens_third_tl \g_helvens_user_tl

\tl_show:N \l_helvens_third_tl

The output is

> \l_helvens_third_tl=macro:
->baz.

as expected. You're easily convinced that no expansion of the items in the token list is performed.

A possibly slower, but in my opinion conceptually better, routine is the following, which is much more general as it allows an arbitrary number of items in the token list.

\RequirePackage{expl3}
\ExplSyntaxOn
\tl_new:N \g_helvens_user_tl
\tl_gset:Nn \g_helvens_user_tl { { foo\frac } { bar\vec } { baz\sqrt } }
\tl_new:N \l_helvens_third_tl
\seq_new:N \l_helvens_temp_seq

% #1 = original token list
% #2 = new token list
% #3 = item to extract
\cs_new_protected:Npn \helvens_get:NNn #1 #2 #3
 {
  \seq_set_split:NnV \l_helvens_temp_seq { } #1
  \tl_set:Nx #2 { \seq_item:Nn \l_helvens_temp_seq { #3 } }
 }

\helvens_get:NNn \g_helvens_user_tl \l_helvens_third_tl { 3 }

\tl_show:N \l_helvens_third_tl

The output is

> \l_helvens_third_tl=macro:
->baz\sqrt .

which shows that no wrong expansion is performed.