# LaTeX3: How to get reliable control over the level of expansion?

Say you want to write LaTeX3 code that manages user data. Sometimes you want to process that data, but other times you just want to store it, move it around and return it unchanged (when developing a data-structure, for instance).

In the latter case you often need to expand down to the user-provided data exactly, but no further, since such data is not always meant for typesetting. Users should be able to store functions inside your data-structure and be sure that when returned, they'll still behave identically.

The problem is, many LaTeX3 functions give no guarantees about the number of expansions necessary to get a specific result.

For example, I had a token list variable \x containing three consecutive brace-groups with user-data. I wanted to put the content of the third group into a variable \y. This is the code I came up with:

\exp_args:NNNo \exp_args:NNo \tl_set:No \y {\exp_last_unbraced:No \use_iii:nnn \x}


I found out that \exp_last_unbraced:No requires 2 expansions and \use_iii:nnn requires one. But this will only work until the implementation of either of those functions silently changes to require a different number of expansions. Then the above code might use one too many or one too few.

How can I get more reliable control over this kind of expansion? I imagine it might have something to do with \exp_not:n and family, but it would be very helpful to have a guide to their proper use.

Edit: Rephrased Question

As observed by existing answers, the answer is of course not to rely on an exact number of expansions. So my question would be better phrased as:

Is there a generally recommended way of 'handling' data that makes it easier to distinguish between expansion until the original level and expansion beyond the original level?

Even before asking the question, I imagined it might have something to do with packaging the data in \exp_not:n {-}. Then if you do an :x expansion, you get exactly the right data back. The problem is, :f expansion, :c expansion, :v expansion, etc. will happily go past this 'barrier' and eat into the data:

\ExplSyntaxOn
\cs_generate_variant:Nn \tl_to_str:n {x}
\cs_generate_variant:Nn \tl_to_str:n {f}

\tl_new:N \l_external_tl
\tl_new:N \l_data_tl
\tl_new:N \l_internal_tl
\tl_set:Nn \l_external_tl  {\l_data_tl}
\tl_set:Nn \l_data_tl      {\exp_not:n{\l_internal_tl}}
\tl_set:Nn \l_internal_tl  {too~far}

\noindent\tt
\tl_to_str:x {\l_external_tl}\\  %  \l_internal_tl  %  good
\tl_to_str:f {\l_external_tl}\\  %  too far         %  bad
\ExplSyntaxOff


So my latest idea is a different kind of 'barrier'. Put the data in a token list variable with a unique csname. And then put only the csname in your internal structures. Then no power in the world can expand the data until a :c-related expansion is used. Even better, simply use a :v expansion to get exactly to the data and no further:

\ExplSyntaxOn
\cs_generate_variant:Nn \tl_to_str:n {x}
\cs_generate_variant:Nn \tl_to_str:n {f}
\cs_generate_variant:Nn \tl_to_str:n {v}

\int_zero_new:N \g__barrier_int
\cs_new_protected:Nn \tl_set_barrier:Nn {
\int_gincr:N \g__barrier_int
\tl_set:cn {barrier(\int_use:N\g__barrier_int)_tl} {#2}
\tl_set:Nx #1 {barrier(\int_use:N\g__barrier_int)_tl}
}

\tl_new:N \l_external_tl
\tl_new:N \l_data_tl
\tl_new:N \l_internal_tl
\tl_set:Nn         \l_external_tl {\l_data_tl}
\tl_set_barrier:Nn \l_data_tl     {\l_internal_tl}
\tl_set:Nn         \l_internal_tl {too~far}

\noindent\tt
\tl_to_str:x {\l_external_tl}\\  %  barrier(1)      %  good
\tl_to_str:f {\l_external_tl}\\  %  barrier(1)      %  good
\tl_to_str:v {\l_external_tl}\\  %  \l_internal_tl  %  good
\ExplSyntaxOff


This way you can pile on fully-expandable operations to your hearts content and just do a :v expansion to get to the data. It's like pointer-redirection.

I will probably write a small package to better facilitate this and use it for my own LaTeX3 programs, unless someone can give me a better option. Let me know what you think.

-
Not an answer, but egreg's point in his answer is correct: if you are thinking 'I need X expansions' then you are almost certainly missing the point of the L3 approach. Apart from at the next-to-lowest level (stuff build just above primitives/:D functions), the idea is that you should really only need to know that you want either 'the stored tokens' or 'the stored value'. Even :o expansion is really meant mainly for low-level code bootstraping the kernel itself and in places where we need performance. –  Joseph Wright Mar 10 '13 at 22:16
@JosephWright I understand that this is the goal. I have just rephrased my question: How to avoid having to 'need X expansions' and still get where you want to go. –  mhelvens Mar 11 '13 at 15:28
I get the feeling you are expecting something from f and x expansion other than what you should expect for a variable. (c-type expansion must give a list of tokens that can generate a name, so that case is different anyway.) You example seems very odd: can you provide a 'real world' case where the currently-available data structures/expansion controls do not work appropriately? –  Joseph Wright Mar 11 '13 at 17:06
@JosephWright There are probably zero such examples. But the point is that you have to figure out which type of expansion control is appropriate in each separate case, whereas the barrier technique always does what you need in a predictable way (for some version of 'always'). It allows you to indiscriminately pile on (fully-expandable) operations (treat L3 like a 'real' programming language). You can then do a :x to process the operations or a :v to be left with only the data. –  mhelvens Mar 11 '13 at 18:34
I get the feeling you are expecting the wrong thing from f-type expansion. It's supposed to give fully-expanded tokens, so any variable use inside such an expansion should 'use' the content of the variable in some way. A standard example is forcing evaluation of an expression by f-type expanding \int_eval:n. Then again, without an example of what you really need to do I wonder if I miss something (I have no programming background, so am simply used to working with TeX). –  Joseph Wright Mar 12 '13 at 8:45

As @egreg already said you are probably using the wrong data type as a starting point. The idea behind expl3 is to avoid using expansion based programming within your programs altogether (or more precisely limit it to very well-defined places).

so the "programming dicipline" would be something like

1. define how the external representation of the user data looks like that you are interested in (in your example 3 brace groups, say)
2. define an internal data structure for the user data that you want to manipulate
3. define a transformation process from external to internal
4. then use only the internal structure for manipulation and provide the necessary mutator functions for your data structure (unless you use an existing one that already has the tools like prop)

Now clearly there is some expansion necessary at some point, e.g., the user input might get scanned into some tl variable and you need to get it out from there to put it into your internal data structure but this step 3. should happen preferably once and could then use simple expansion rules, e.g.,

   \exp_after:wN \helvens_store_user_data:nnn \x


or if the data come in via some document-level argument via

   \helvens_store_user_data:nnn #1   % with #1 being expected to be 3 brace groups


but neither the \x nor the #1 should be long-living and passed around in your code unless you declare that \xis a variable of your data structure type and then you should have some manipulation functions that go:

   \helvens_retrieve_third:NN \x\y


Yes, such a function would need to do some low-level expansion (due to the choice of your data structure) but those could be handled with the tools available (relying on the documented expansions that certain functions need), eg.

   \cs_new_protected:Npn \helvens_retrieve_third:NN #1 {
\exp_after:wN \__helvens_retrieve_third:nnnN #1
}
\cs_new:Npn \__helvens_retrieve_third:nnnN #1#2#3#4 {
\tl_set:Nn #4{#3}
}


and similar for any other functions that you want/need to manipulate your data structure.

But please don't go for something like

   \exp_args:NNNo \exp_args:NNo \tl_set:No \y {\exp_last_unbraced:No \use_iii:nnn \x}


in the middle of your programming code to avoid setting up data structure manipulation functions.

Having said all this ... you make some valid points here and we should

• better document this concept/approach
• document for those expansion functions that you reasonably need for setting up data structures how their expansion behavior is precisely; it is only a small set that should be used for this and most of not all of them are of type 1 expansion

In one of the comments it was mentioned that such a solution as given by egreg or myself would "perhaps" not work in some situations. If so I would suggest to provide those scenarios as additional questions. In our opinion this approach should be generally applicable.

-
Thanks for this answer! I actually agree with you on every point. But your answer is based on a misunderstanding of my original question. I have now rephrased my question, which was purely about how to ensure reliable expansion within the implementation of my data-structure (and the three-brace-group thing was a very localized example). I also suggest a solution meant to be quite 'general'. I would like to get your opinion on that. –  mhelvens Mar 11 '13 at 15:45

I'd say that you're using the wrong data type, in this case. However, you don't need to control how many expansions you need to do:

\RequirePackage{expl3}
\ExplSyntaxOn
\tl_new:N \g_helvens_user_tl
\tl_gset:Nn \g_helvens_user_tl { { foo } { bar } { baz } }
\tl_new:N \l_helvens_third_tl

\cs_new_protected:Npn \helvens_get_third:NN #1 #2
{
\exp_last_unbraced:NNV \__helvens_get_third_aux:Nnnn #1 #2
}

\cs_new_protected:Npn \__helvens_get_third_aux:Nnnn #1 #2 #3 #4
{
\tl_set:Nn #1 { #4 }
}

\helvens_get_third:NN \l_helvens_third_tl \g_helvens_user_tl

\tl_show:N \l_helvens_third_tl


The output is

> \l_helvens_third_tl=macro:
->baz.


as expected. You're easily convinced that no expansion of the items in the token list is performed.

A possibly slower, but in my opinion conceptually better, routine is the following, which is much more general as it allows an arbitrary number of items in the token list.

\RequirePackage{expl3}
\ExplSyntaxOn
\tl_new:N \g_helvens_user_tl
\tl_gset:Nn \g_helvens_user_tl { { foo\frac } { bar\vec } { baz\sqrt } }
\tl_new:N \l_helvens_third_tl
\seq_new:N \l_helvens_temp_seq

% #1 = original token list
% #2 = new token list
% #3 = item to extract
\cs_new_protected:Npn \helvens_get:NNn #1 #2 #3
{
\seq_set_split:NnV \l_helvens_temp_seq { } #1
\tl_set:Nx #2 { \seq_item:Nn \l_helvens_temp_seq { #3 } }
}

\helvens_get:NNn \g_helvens_user_tl \l_helvens_third_tl { 3 }

\tl_show:N \l_helvens_third_tl


The output is

> \l_helvens_third_tl=macro:
->baz\sqrt .


which shows that no wrong expansion is performed.

-
Thanks for the effort. :-) You're obviously right for this example. But I should point out that it was just meant as an example to a more general problem. There are other situations for which this kind of solution might not work. I was hoping for a... programming discipline to avoid this problem altogether. I can imagine various avenues this discipline might take. Such as \exp_not:n or perhaps 'barriers' using csnames, so you'd need to do a :c expansion to get through. –  mhelvens Mar 10 '13 at 21:32
@mhelvens My point is that you shouldn't rely on any number of required expansions to achieve your result, unless the number is clearly stated in the documentation (it is for some of the functions). –  egreg Mar 10 '13 at 21:50
Indeed. But I already drew that conclusion before I asked the question. Referencing the 'exact number of expansions' was only part of the problem statement. I should have phrased my question more clearly. And I'll do so tomorrow. –  mhelvens Mar 10 '13 at 21:56
In the first code snippet all functions should be protected. In the second, you can use \seq_set_split:NnV \l_helvens_temp_seq { } #1 rather than mapping through the token list. Then indeed \seq_item:Nn is the right tool to use as it wraps its result in \exp_not:n before outputting it. –  Bruno Le Floch Mar 10 '13 at 23:22
@BrunoLeFloch Thanks for the suggestions –  egreg Mar 10 '13 at 23:29