Let's assume someone who has used LaTeX for some years, and is starting to write his/her own macros (that are beyond the level of \newcommand{\MyVector}[1]{\mathbf{#1}}, performing more complicated tasks, involving maybe temporary variables, arithmetics, string manipulations etc.).

Is it a good advice that he/she should directly learn and apply expl3? Avoiding understanding – and learning by heart – the whole TeXbook, LaTeX's article.cls, not learning the art of \expandafter\expandafter\expandafter, the differences among {, \begingroup, and \bgroup, the differences among \if, \ifnum, \ifcase etc., commands like \@namedef, \@ifnextchar, \@gobble, \@gobbletwo, where space characters count and where they don't etc.

To me it seems that, during the construction of expl3, the team started with collecting all their frustrations regarding TeX and LaTeX, and aimed at minimizing these with expl3. Has this been successful to the level that one can start the life of serious macro writing directly with expl3?

  • 5
    You will need to understand some TeX concepts: we can't avoid tokens and expansion. I guess beyond that, this looks quite opinion-based.
    – Joseph Wright
    Jul 21 '20 at 9:56
  • Thank you, @JosephWright. "Opinion-based": If you say this then it's not very encouraging…
    – TFulop
    Jul 21 '20 at 11:11
  • I mean that as a mod I can see an argument here: the answers are unlikely to be technical, as there are more about what people are happy with. On the other hand, there are technical elements (for example coverage of concepts): I'm not going to close the question!
    – Joseph Wright
    Jul 21 '20 at 11:37
  • I'm optimistic: this may not only be a matter of taste but some more-or-less quantifiable experiences of those using expl3 for a long time. 1. How easier you feel you can proceed with expl3 than without it? 2. How many percent of the problems you can solve in expl3 (naturally an estimate only)?
    – TFulop
    Jul 21 '20 at 11:44
  • 1
    "the art of \expandafter\expandafter\expandafter": When I play with expl3, the lack of necessity often robs me of the opportunity to apply the art of \romannumeral0-expansion. ;-) Jul 21 '20 at 23:15

The broad base of programming tasks is covered by expl3 and so it is possible to use it for a large number of tasks. For example, my package siunitx is written entirely in expl3, and there are very few places I have to grab some LaTeX2e or plain TeX concept. (Those tend to be at the interface, for example using LaTeX2e document commands, or one or two very limited places there are no expl3 interfaces for TeX primitives.)

You will need to understand some basic TeX ideas to use expl3, most notably

  • TeX works with tokens not characters/strings, which means we need to worry about category codes
  • TeX arguments are normally 'balanced text' (things in braces or single tokens), unless they are explicitly set up to be delimited
  • TeX works by expansion, but not everything is 'expandable'
  • Scope is created explicitly using groups, not implicitly by 'functions'

(Over time, the LaTeX team do hope to document all of this within the expl3 documentation.)

With that background, you should be able to tackle a wide range of programming tasks in expl3. You may need some 'glue' where for example you need to use a LaTeX2e concept, e.g. package loading, but this will typically be a very small part of your code. In particular, you are very unlikely to need complex expansion control, unless you are after high performance or very specific needs.

It's important to state that expl3 is broadly stable. There are still deprecations, and you may need to make adjustments over time, but these are increasingly rare. There is a formalised mechanism for deprecation and informing programmers of changes.

  • 1
    TeX working with tokens and there being active characters also implies that a) you can't rely on a single character token yielding a single letter in the document. b) you can't rely on a single character of the .tex-input-file yielding a single character-token: E.g., using (n)german-babel-shorthands you can write ü as "u which yields active " trailed by the letter u. Processing utf8-encoded files on an 8bit TeX-engine using the inputenc-package implies that single utf8-characters of the .tex-input-file get tokenized as several character-tokens whereof the leading ones are active... Jul 22 '20 at 10:09
  • Systematic, deep, clear – thanks a lot, @JosephWright.
    – TFulop
    Jul 22 '20 at 10:31
  • @UlrichDiez All true but unless one starts decomposing tls contained text, it's not an issue.
    – Joseph Wright
    Jul 22 '20 at 10:46
  • @JosephWright As can be seen with the listings-package which does a lot of examination of contained text, it indeed is an issue then. ;-) I think decomposing is not the only source of possible issues: E.g., using an 8-bit-engine and applying \uppercase/\lowercase-primitives to a sequence of several character-tokens denoting one multibyte-character/utf8-character does not imply decomposing and may yield a result differing from what you get when applying \uppercase/\lowercase-primitives with an utf8-engine where one character-token is sufficient for denoting the character in question. Jul 22 '20 at 12:23
  • 2
    @UlrichDiez listings is quite a special case, but does point to the need for additional text-processing tools (at present most things are focussed at the token list level). For case changing, there are no interfaces exposing \lowercase/\uppercase: expl3 has \text_lowercase:n, etc. for text and \str_foldcase:n, etc. for programmatic strings. For the text version, UTF-8 is accounted for. (For the string version, that's not the case at present but that's largely as programmatic strings likely shouldn't 'expect' active-char like behaviour.)
    – Joseph Wright
    Jul 22 '20 at 12:51

[I decided to remove my excessive comments and to provide an answer instead.]

If I got you right, you wish to know if it is good for someone who wishes to accomplish more or less sophisticated programming-tasks by means of TeX to directly lay the focus on expl3 rather than beginning by gaining in-depth knowledge of plain-TeX and primitives and the LaTeX 2ε-kernel.

This also depends on the purposes behind these programming tasks.

If you wish to just write some more or less sophisticated macros for private usage, you may be happy when getting it done somehow, e.g., by means of expl3 if that is what you are already familiar with.

If you wish to publish code which is intended to extend a given framework, you need to be familiar with that framework and with all other frameworks you may use for writing your code:
Writing a package for LaTeX 2ε by means of using expl3 in package-mode implies the need of being familiar both with LaTeX 2ε and with expl3.

In case first attempts at writing own macros using expl3 do not work out as intended, you may wish to retrace things for finding the reason. Hereby you may easily and quickly reach the point where you wish to retrace what an expl3-macro defined in the expl3-kernel does exactly.
As something was used for implementing the expl3-kernel, you then have reached a point where you need to be familiar with that "something". That "something" basically is some TeX-engine with primitives "renamed".
So you still need to be familiar with the basic concepts ("digestion processes": reading and tokenizing .tex-input-files, expansion, assignments, glue and boxes, output-routines etc) that apply to all TeX-engines.

[Explanatory insert:]

I just wrote:

In case first attempts at writing own macros using expl3 do not work out as intended...

I don't intend to indicate that expl3 (or whatever other framework) is not reliable/is unstable or the like.
When I started learning TeX/LaTeX 2ε/expl3 my attempts at writing own macros very often did not work out.

The reason was not the "framework". The reason was that in my beginnings I didn't know enough about that "framework" (only superficial knowledge conveyed via some nice tutorials, but no in-depth knowledge) and therefore tried to use things in ways which were not intended by the implementers who wrote the "framework". In order to gain knowledge about the "framework" I needed to play around and trace things ("looking" at them via \show) and grasp how they work myself.

Back then I received great help by the regulars of the usenet-newsgroups comp.text.tex and de.comp.text.tex.

[End of explanatory insert.]

On the one hand expl3 is designed with the goal that you don't have to worry about every obscure expansion-sneak when programming.
On the other hand expl3 is designed to remind you of concepts like tokenization and expansion all the time, again and again, ad nauseam, because with expl3 the name of a non-user-level-macro already tells you what arguments that macro processes and by means of which tokenization/expansion/whatsoever-methods each argument of the macro will be pre-processed before it gets applied to the macro-definition's replacement-text.
I take this for a good thing because it can be very helpful when in case of error-messages retracing things be just looking at the code: You can more easily see that a macro doesn't get the type of argument which it should get. E.g., a macro getting a normal argument while "expecting" an argument that got tokenized under verbatim-category code régime. E.g., a macro getting the name of a control-sequence while expecting the control-sequence-token. ...

It is also a bit like cars: In order to handle it well, you have to know to a certain extent how it works.
My experience is that with expl3 reading the user documentation usually is sufficient— provided you read it very very carefully and understand the references to basic TeX concepts.
Nevertheless you may reach a point where you are interested in expl3's implementation...

If you like carefully reading user-documentations in the way in which a lawyer investigates every word/phrase of a contract for pitfalls, probably needing to do a bit of research within other sources of information like the TeXbook regarding the meaning of TeX-related terminology and concepts, then in my opinion expl3 is a good choice.
If you like to learn by reading code, then it also depends on what macro-collections etc you are already familiar with.
I think if you start into the "TeX-world" by learning expl3, then the learning curve will be very steep and you will be faced to references to basic TeX-concepts in an extent which sooner or later leads to you being able to cope with code written in LaTeX 2ε or plain-TeX as well. ;-)

To be honest: To me retracing expl3-code is a nightmare and it takes me long time. This is mainly because I am not familiar with/used to it. Thus I think: If you wish to quickly reach a stage where this is not a nightmare to you, then start learning expl3 as soon as possible. ;-)
I think it is also a matter of training: In my beginnings when I wasn't familiar at all with TeX-programming and the look of LaTeX 2ε-code, retracing LaTeX 2ε-code (e.g., by reading source2e.pdf) was a nightmare to me, too. ;-)

Currently/nowadays you will nevertheless still be faced to a lot of code not written in expl3. If you wish to be able to trace such things, you need to be familiar with programming in plain TeX and LaTeX 2ε as well.

  • 1
    Debugging expl3 needs something other than \tracing because the log is just so crowded with internal details. It's not just you, or your unfamiliarity. Jul 21 '20 at 23:31
  • @DonaldArseneau Depends on your approach ;) I find searching through the log for my internal macros, so I can see their arguments, works pretty well. (I tend to make sure all of my internals receive exactly the list of arguments 'expected' by their arg spec, including appropriate expansion.)
    – Joseph Wright
    Jul 22 '20 at 8:38
  • @DonaldArseneau All that said, Bruno has made suggestions in the past: I am sure there are better ways
    – Joseph Wright
    Jul 22 '20 at 8:38
  • I’ll raise on the team list
    – Joseph Wright
    Jul 22 '20 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.