3

There are a lot of functions provided by the expl3 programming environment that in interface3.pdf are marked with a black asterisk (fully expandable) or with a white asterisk (restricted expandable).

Why are so many functions marked fully expandable without providing official information about the amount of expansions needed for obtaining the result?

Well, with some fully expandable tail recursive things where means of expansion control like \exp:w/\exp_end: are not applied in the implementation, both the amount of recursions and the amount of expansions needed for obtaining the result depend on arguments provided by users. Therefore exact information about the amount of expansions needed for obtaining the result cannot be provided there.

But there are a lot of functions where this is not the case and where exact information about the amount of expansions needed for obtaining the result could be provided.

How, with such fully expandable expl3 functions, to know for certain in expansion contexts the moment when the result is there?

E.g., I'd like to know the amount of expansions that need to be triggered for obtaining the result of \str_tail:n.

In expansion contexts you can't use x-expansion as that itself is not expandable.
You can't safely just use f-expansion as that might remove another leading space from the string.
If the fully expandable function in question was not \str_tail:n but something where the arguments themselves are not strings of unexpandable explicit character tokens, then e-expansion might trigger more expansion than is wanted. Besides this, with engines where the \expanded-primitive is not available, e-expansion is expensive.

In the case of \str_tail:n one could get the idea of looking into source3.pdf and finding out that with the current implementation four expansions are needed:

\cs_new:Npn \str_tail:n #1
  {
    \exp_after:wN \__str_tail_auxi:w
    \reverse_if:N \if_charcode:w
    \scan_stop: \tl_to_str:n {#1} X X \s__str_stop
  }
\cs_new:Npn \__str_tail_auxi:w #1 X #2 \s__str_stop { \fi: #1 }

Expansion 1 gets you the replacement text of \str_tail:n.
Expansion 2 does the \exp_after:wN-thingie which ends when the result of the \if_charcode:w-test is evaluated and hereby the first token of the string is used up as argument for comparing its category code to the category code of \scan_stop:.
Expansion 3 expands \__str_tail_auxi:w which prepends \fi: for matching up \if_charcode:w and keeps the category-11-X-delimited argument and removes the \s__str_stop-delimited argument.
Expansion 4 removes \fi:.

But looking into source3.pdf is not a way of obtaining information about official details of the implementation.

In case the internal implementation changes, the amount of expansions needed for obtaining the result of \str_tail:n might change as well.

I don't see a general method with a fully expandable function for ensuring that you get the right moment for surrounding with some other tokens the tokens that form the result of that function other than knowing the amount of expansions after which the result of the function is there.
When using \exp:w for triggering expansion of a fully expandable function provided by the expl3 programming environment until the result of that function is there, I don't see a generic method for ensuring that \exp_end: is carried out at the right moment in time for stopping \exp:w-expansion which can do without knowing the amount of expansions after which the result of that function is there.

So I think with fully expandable functions, this information should be provided as an official bit of information.

How many expansions are needed until the result of functions \exp_args:... or \use_i:nn is there?
That information is not officially provided in interface3.pdf.
Can I safely rely, e.g., on \use_i:nn always requiring exactly one expansion without somebody whining because I deduce information from internal implementation details instead of only relying on facts that are documented officially?
What should cause me to rely on consistency across expl3 releases in aspects that are not documented officially?
(The LaTeX distribution that comes along with my Linux distribution differs so much from TeX Live 2024... )

6
  • 3
    The number is not documented by design as it is an implementation detail and may (and historically has) change. If the number of expansions was a documented part of the interface it would severely limit possibilities for re-implementation.refactoring. For an extreme example consider e arguments with and without an \expanded primitive. Commented Apr 17 at 18:45
  • 1) With any fully expandable function adding expansion-control in terms of \exp:w/\exp_end: or in terms of combining \use:e and \exp_not:n would be possible so that exactly two expansions were needed. But this would be expensive in terms of semantic nest. 2) A lot of changing goes on these days, so don't rely on anything not implemented by yourself to be consistent! 3) Why to rely on official information not to be subject to change? 4) Why to rely on users not redefining things used in your code and which your code relies on? 5) There is no long-term safety in (La)TeX! Commented Apr 17 at 18:54
  • 2
    Not directly relevant to an answer: \expanded is nowadays a required primitive, so the efficiency of emulation of e-type does not occur
    – Joseph Wright
    Commented Apr 17 at 19:08
  • 1
    @UlrichDiez We very rarely make changes to the documented API for expl3 functions other than to correct errors - I am not sure what you mean
    – Joseph Wright
    Commented Apr 17 at 19:10
  • 1
    @JosephWright A lot of development and change takes place and is planned with the LaTeX 2e-kernel.And with hyperref. So don't rely on any internal bit of implementation to remain the same. That's what I mean. Official docu. of expl3 didn't change much in the past, but I don't see any warranties for the future here. Maybe the members of the LaTeX team change some day and the people in charge then decide to introduce changes to the official interface? When it comes to the question of what the future will bring, despite all well-intentioned promises, there are no real guarantees. That's life. ;-) Commented Apr 17 at 19:32

1 Answer 1

4

The documentation situation is by-design. There are very few (low-level) functions where the number of expansions is important, and where the team or others have needed to know/rely on it: these are documented. In other cases, the use of e- or f-type expansion is typically sufficient. For example, \str_tail:n necessarily produces a string - this is therefore safe in an e-type context, so there is no need to document how many expansions are required.

More generally, if a function takes document tokens then you can't use e-type expansion, but in general stick to \protected@edef as it works with LaTeX2e robust commands - expl3 doesn't generally do this. For out-and-out text, \text_expand:n does something similar to \protected@edef but doesn't work well with non-textual expansion. Either way, these approaches allow expansion of expl3 functions whilst preserving LaTeX2e fragile material (e.g. \textbf or similar).

Documenting the number of expansions required places a restriction on any code changes: as far as possible we do not do this - the restrictions are only those which form part of the API. If you have a clear use case which seems to require such API stability, log an issue and ask for a (documentation) change.

6
  • @Cattleya I realise that \str_tail:n was an example but it was the one given so I could give an explanation of why the expansion detail is not needed there. As I say, there may be cases where the detail is required, and these can be added to the docs - we do cover it for some functions when it is critical.
    – Joseph Wright
    Commented Apr 17 at 19:18
  • @Cattleya I am not sure what you mean about declaring things obsolete
    – Joseph Wright
    Commented Apr 17 at 19:18
  • @Cattleya You are officially told that internal implementation details may be subject to change. More honesty is not possible. In rare situations where you need to know the exact amount of expansions but are not officially told, don't directly use the official interfaces but define your own. Often you can create your own interface where (some) arguments are wrapped into some expansion-prevention/protection and then passed on to the official interfaces so that e-expansion expanding too much is not an issue. Commented Apr 17 at 19:45
  • @JosephWright Please don't take this as a complaint about the team's work, but I have a hard time understanding how you can fail to know what is even meant. Things in l3obsolete.txt? Indeed, even that file changes format so that a script written to parse it cannot be relied upon to work without modification following an update. I don't know which were experimental and which not and I'm sure you have good reasons for removing them. Maybe you even have good reasons for altering the reporting format when you break my scripts. But I don't see how you can fail to understand what is meant.
    – cfr
    Commented Apr 18 at 4:29
  • [Actually, I think none were experimental? Only official ones get into the file?]
    – cfr
    Commented Apr 18 at 4:40

You must log in to answer this question.