# LaTeX3 versus pure Lua

Reading this site for the last nearly two years I've learned a lot about LaTeX3 (which I had previously understood to be a defunct pipe dream) and LuaTeX (of which I was previously totally unaware). As I have said before, TeX itself is a bad programming language, though a good typesetting engine, but even as a typesetting engine, it is hobbled by its limitations as a language. What I mean by both of these, in short, is that it does not provide an easy way to manipulate its own representations of data. Others have also had this problem, and sometimes it can be solved with LuaTeX, which opens up a lot of the internals to the programming environment.

Now, as I understand it, LaTeX3 also provides a lot of programming tools within TeX, with the aim of making package creation and maintenance more robust. Joseph Wright has explained this in at least one place (and therefore probably many more). These are "algorithmic" tools, while the direction of LuaTeX development appears to be "data structure" tools. Of course, Lua is also a general-purpose programming language and as such provides perfectly good algorithmic tools as well.

My question:

Why use LaTeX3 programming over Lua, when LuaTex is available?

I understand that LuaTeX is very much experimental now, but then, so is LaTeX3. I also understand that as a result, LuaTeX is not necessarily used by anyone "ordinary", but then, it is intended as the ultimate successor to pdfTeX, which is now the default engine in the major TeX distributions. For all the reasons that TeX is a terrible programming language, LaTeX3 will still suffer from irritating expansion issues, lack of arithmetic capabilities, poor (or at least poor man's) variables, and so on. However, Knuth himself has written that he would have used a better language if one had been available. Now that "we" are inserting Lua into the TeX executable it appears that this alternative has been identified (ironically, that link is about PyTeX, based on Python).

Now that \directlua is feasible, I don't see why it is not better in every way simply to make all "function" macros (those that do not directly store text) hook into that one primitive and then do all their programming in Lua. Loops, variables, recursion, and so on can all be done in a comprehensible way. A lot of effort in the development of LaTeX3 appears to be in managing the programming language and its use. So, I ask again:

Why use LaTeX3 to program at all? Why not turn its development into the development of a LaTeX-style document design library for LuaTeX, written in Lua?

This suggests a related question to me:

Is the separation of LuaTeX and LaTeX3 a result (or at least an artifact) of the non-communication among developers that Ahmed Musa described in his comment to this answer? What kind of cooperation is there between these two projects to reduce duplication of effort?

(An even more fantastical notion is to implement every primitive, except \directlua itself, in terms of Lua and various internal typesetting parameters, thus completely divorcing the programming side of TeX from the typesetting side. Then TeX itself would be a "macro package" on top of LuaTeX and one would not be bound to its strange design choices. An answer to why that should or should not be done might be too long for me to request, however.)

I don't personally think that this is too bloggy for this site, and I hope others agree. At least, given the confluence of expertise here, it seems to be the best place to ask the question.

I have accepted Frank Mittelbach's answer, though all the answers are extremely informative, because I think it hits all the main points most directly. One thing that neither it nor Joseph Wright's answer quite manages to come out and say directly is that ConTeXt does do the Lua-programming thing at least to some extent, and it does not solve all algorithmic problems because LuaTeX does not actually rewrite the underpinnings of TeX so much as stitch in a Lua layer that can interact with them.

I guess the one-line answer to my question is that my expectations of LuaTeX are much more grand than its reality, much like my "fantastical" idea.

• interesting question. i hope that answers address the concept of stability that is the underpinning on which production shops depend. neither luatex nor latex3 hold any promise of meeting that goal in the near future (say in two years or less), and it has been said by the main luatex developer (unless i've misunderstood) that reproducibility of line and page breaks is not among their stated goals. – barbara beeton Feb 20 '12 at 19:43
• @barbarabeeton If it's true that luatex isn't necessarily going to reproduce line and page breaks (even if under a "default microtypographic regime"), surely that's a major strategic impediment to adoption for any "legacy house"? – Brent.Longborough Feb 20 '12 at 20:48
• @Brent.Longborough -- yes, of course that's a serious concern. but we have to wait until luatex is declared stable before we can even begin to explore the ramifications. at the moment, it's a wait-and-see situation. all we can do is try to keep a list of what must be tested in order to make a decision. it's in neither our interest nor that of our authors to refuse to adapt to "modern technology", but we can't afford surprises either. that's why i thought this might be a good opportunity to make the rest of the community think about it. – barbara beeton Feb 20 '12 at 21:37
• We have to admit that we sacrifice the holy cow of "our document must look exactly the same in 30 years" already, and there is no need to get upset that LuaTeX does that more visible then currently. We change output formats (DVI -> PDF), we update packages, we switch fonts (cmr bitmap -> type1) and are surprised that the documents don't look the same anymore? If you need true backwards compatibility: use TeX and dvi output. Or keep your old TeX installation. Use virtual machines to keep them running. – topskip Feb 20 '12 at 21:47
• @Brent.Longborough: I have yet to hear of anyone "with a big investment" to invest substantially into the LuaTeX/ConTeXt/LaTeX3 developement. My impression is the opposite: Publishers like Springer are no longer interested in the developement of TeX (if they ever where). If that is true, we don't have to care more for their interests than they do. AFAIK the only one investing substantially into ConTeXt are Hans Hagen (time and money) and the TeX user groups (time). – Martin Schröder Feb 22 '12 at 11:02

## 5 Answers

(I guess I could be called a member of one of the teams ;-) this is my view)

I thought of staying out of this debate, but perhaps some words of clarification or, let's say, some thoughts are in order after all.

LaTeX3 versus pure Lua

First of all this is the wrong question imho: LaTeX3 has different goals to LuaTeX and those goals may well be still a defunct pipe dream, but if so they are unlikely to be resolved by pure Lua either.

So if one wants to develop an argument along those lines then it should be more like "Why does LaTeX3 use an underlying programming language based on eTeX and not on LuaTeX where a lot of functionality would be available in a "simpler" way?"

But LaTeX3 is really about three to four different levels

• underlying engine
• programming level
• typesetting element layer
• designer interface foundation layer
• document representation layer

See for example my talk at TUG 2011: http://www.latex-project.org/papers/

Here is a sketch of the architecture structure:

The chosen underlying engine (at the moment is any TeX engine with e-TeX extension). The programming level is what we call "expl3" and that is what I guess you are referring to if you say "LaTeX3" (and I sometimes do that too). However, it is only the bottom box in the above diagram. The more interesting parts are those that are above the programming level (and largely a pipe dream but moving nicely along now that the foundation on the programming level is stable). And for this part there is no comparison against Lua.

Why use LaTeX3 programming over Lua, when LuaTeX is available?

To build the upper layers it is extremely important to have a stable underlying base. As @egreg mentioned in chat: compare the package situation in 2.09 to the package situation in 2e. The moment there were standard protocols to build and interface packages the productivity increased a lot. However, the underlying programming level in 2e was and is still a mess which made a lot of things very complicated and often impossible to do in a reliable manner. Thus the need for a better underlying programming layer.

However, that programming layer is build on eTeX not because eTeX is the superior engine (it is compared to TeX but not with respect to other extensions, be it Lua or some other engine) but because it is a stable base available everywhere.

So eTeX + expl3 is a programming layer that the LaTeX3 team can rely on of not being further developed and changed (other than by us). It is also a base that is immediately available to everybody in the TeX world with the same level of functionality as all engines in use are implementing eTeX extensions.

Any larger level of modifications/improvements in the underlying engine is a hindrance to build the upper layers. True, some things may not work and some things may be more complicated to solve but the tasks we are looking at (well I am) the majority are very much independent of that layer anyway.

To make a few examples:

• good algorithms for automatically doing complex page layout aren't there (as algorithms) so Lua will not help you here unless somebody comes up with such algorithms first.

• something like "coffins", is thinking about approaching "design" and the importance here is how to think about it now how to implement it (that comes second) -- see Is there no easier way to float objects and set margins? or LaTeX3 and pauper's coffins for examples

Having said that, the moment LuaTeX would be stable similar to eTeX (or a subset of LuaTeX at least) there might well good reasons for replacing the underlying engine and the program layer implementation. But it is not the focus (for now).

Why use LaTeX3 to program at all? Why not turn its development into the development of a LaTeX-style document design library for LuaTeX, written in Lua?

Could happen. But only if that "LuaTeX" would no longer be a moving target (because LaTeX3 on top would be moving target enough).

Side remark: @PatrickGundlach in his answer speculated that this answer that the LaTeX3 goal is backwards compatibility. Wrong. The same people that are coming down on you very strong about compatibility for LaTeX2e have a different mindset here. We do not believe that the interesting open questions that couldn't get resolved for 2e could be resolved in any form or shape with LaTeX3 in a document-compatible manner.

Input compatible: probably. But output compatibility for old documents, no chance if you want to get anything right.

But in any case, this is not an argument for or against implementing the ideas we are working on one day with a LuaTeX engine.

Is the separation of LuaTeX and LaTeX3 a result (or at least an artifact) of the non-communication among developers that Ahmed Musa described in his comment to this answer? What kind of cooperation is there between these two projects to reduce duplication of effort?

As I tried to explain, there is not much overlap in the first place. There is much more overlap in conceptual ideas on the level ConTeXt viz. LaTeX.

An even more fantastical notion is to implement every primitive, except \directlua itself, in terms of Lua and various internal typesetting parameters, thus completely divorcing the programming side of TeX from the typesetting side.

That brings us to a completely different level of discussion, namely is based on LuaTeX, or anything else for that matter, a completely different approach to a typesetting engine possible? That is a very interesting thought, but as @Patrick explained it isn't done with leaving TeX to do the typesetting and do everything else in a different language. So far such concepts have failed whether it was NTS or anything else because fundamentally (in my believe) we haven't yet grasped how to come up with a successful and different model for the TeX macro approach (as ugly as it might look in places).

• Just the kind of reply I was hoping for. To clarify: I previously believed it to be a defunct pipe dream, based on what I read on the internet. Also, you have correctly understood that this is a question about the programming layer and not the typesetting and above layers (though both can be called "LaTeX3", as you say). Finally, I should say that I'm not trying to start a flamewar, so I hope that anyone who can answer, will feel comfortable doing so. – Ryan Reich Feb 21 '12 at 0:42
• What I had in mind when wrote "backward compatibility" is that my "\documentclass{..} .. \section{whatever} .. \end{document} document still gets processed. Not necessarily with exact the same layout. – topskip Feb 21 '12 at 9:36
• @Patrick that is what I call input compatibility. But this is independent of the underlying engine or underlying programming layer used - that was my point. – Frank Mittelbach Feb 21 '12 at 9:57
• @Khaled I fear that doesn't prove anything. If you are the developer of both or very closely aligned then the "hindrance" is smaller but it doesn't go away. One point here is that you don't have your user base being required to sync with several developments in parallel and engine updates do propagate much slower. – Frank Mittelbach Feb 21 '12 at 14:17
• @SeanAllred one shouldn't read too much into it, but in a nutshell: the green blocks are the foundational parts (the programming layer and the support parts for the other layers); the yellow/orange blocks are the configuration data components for document design (design by declaration); the blue is the UI interface layer, ie how objects are externally specified; and the red is simply real world documents that are written in whatever the blue layer provides. There should be some other posts that show this in somewhat different pictures that may make this clearer. – Frank Mittelbach Aug 15 '13 at 16:23

Both Patrick and Frank have provided good answers: I'd like to add a few complementary points which seem too long for comments.

One aspect of expl3 that is important is that it can be used now to program LaTeX2e packages (the 'l3in2e' approach). That is seen for example in my own siunitx package, which I would not want to make LuaTeX-only. A particularly important aspect to this is xparse, which is proving to be popular with a reasonable number of users who probably don't care about the programming internals. This use of LaTeX3 code is dependent on being usable with a good range of the currently-available engines.

A second aspect is that LuaTeX does not mean you need no TeX programming at all. TeX is still there doing the typesetting, but also things like argument grabbing. (On the later, I asked the ConTeXt people about this once as I wondered about the old verbatim-in-arguments issue.) So improved TeX programming is still a good idea, even if you decided to go LuaTeX-only. Now, the nature of the structures you need would doubtless be different in that case, but again you can't simply ignore it.

Finally, there are practical considerations. A lot of the work on expl3 was done well before LuaTeX began, or certainly before it started reaching a more stable state. So to move from expl3 to Lua now would not really cut down on the work we need to do today, which as Frank says is mainly about higher-level issues. A lot of what needs to be done seems to me to be relatively insensitive to the choice of language: it seems unlikely that a switch to Lua would suddenly accelerate development.

(I am not a member of either team, this is my view.)

LaTeX3 is not only about providing a better output routine, but also about providing programming tools such as the famous quicksort algorithm (see l3prg.sty). This is obviously nicer to write in Lua (for most of us). Or consider string manipulation / regular expressions. This could be handled in Lua as well. But strictly speaking, one doesn't need Lua for that.

Why use LaTeX3 to program at all? Why not turn its development into the development of a LaTeX-style document design library for LuaTeX, written in Lua?

As I understand LaTeX3, one of its major goals is keeping backwards compatibility. While other documentation approaches are surely much appreciated in todays world (XML typesetting with CSS formatting for example), they would break existing documents. This is not what the LaTeX3 team wants.

What kind of cooperation is there between these two projects to reduce duplication of effort?

I can't see much communication between the two projects. LaTeX3 is aimed at "working with all engines that have the eTeX extensions". This implies that no engine specific code must be used (that is: functionality that behaves differently under different engines). Therefore there is no obvious need for LuaTeX specific code.

An even more fantastical notion is to implement every primitive, except \directlua itself, in terms of Lua and various internal typesetting parameters, thus completely divorcing the programming side of TeX from the typesetting side.

Lua does not do any macro expansion / replacement, you need TeX for that. It is not possible to delegate all work to Lua, you will need the TeX part of LuaTeX.

On the other hand, both LaTeX and Lua offer control structures such as loops or if/then/else clauses (LaTeX3 even more). Therefore you don't need Lua at all. If you mix control structures from Lua and TeX, the programming can get ugly, because you need to switch back and forth between TeX and Lua. Currently this is only possible with coroutines but may be easier in the future.

LuaTeX's great power in LaTeX lies in the manipulation of the (pre) typeset material.

• The "Lua for the programming" idea certainly resonates. As a programmer (z/Assembler, C, Ruby, Python, Haskell) I must confess I often feel, on looking into the entrails of a LaTeX package, that I am in an alien land. – Brent.Longborough Feb 20 '12 at 22:06
• My intent with "Lua for the programming" is that on the TeX side, you only define control sequences of the form \cs = \directlua{Lua code} or \cs = {plain text}; TeX does the expansion (to exactly one level deep). In addition, one should use none of the TeX control structures; all of that would be done in Lua. You're right that the programming can get ugly; in fact, that's exactly what examples of LuaTeX that I've seen here look like. And the reason is that currently, you have to do half the progrmaming in Tex still. – Ryan Reich Feb 20 '12 at 22:15
• "On the other hand, both LaTeX and Lua offer control structures such as loops or if/then/else clauses (LaTeX3 even more). Therefore you don't need Lua at all." — The point is that you often need much more than just if/then/else. If you look at any of the literally tens of thousands of questions on this site of people having programming problems, you'll see that what are just basic use of data structures and programming paradigms in other languages are unnecessarily convoluted in latex. – Neil G May 5 '15 at 4:55
• What you don't need actually is "macro expansion". So in fact, you don't need tex at all. What you need is typesetting in lua or python or some other language. – Neil G May 5 '15 at 4:56

My comments are from a user and programming perspective, as I am not a member of either Team.

The segmentation of the problem as Frank described it hit the nail on the head and should be the starting point of any critical review and comparison. To this we need to ask the question:

What problem are we solving and how can we summarize it?

Perhaps the answer is as simple as: "from a a set of markup typeset a document, based on a Document Description Language".

From this simple statement one can then start subdividing the problem into smaller parts, both from a user perspective as well as from a programming and algorithmic point of view.

So how are we to mark a document?

The obvious answer is, we need to type the text as text and the markup we need to denote it somehow different. I don't share the view that this can be simplified and shorter than as for example \makeindex even if users long for Markdown or Pandoc (good attempt to standardize markup), but so far this has been proven inadequate for complex commands or layouts. (all)TeX is flexible enough to adapt to such a change, if for example ten years from now markdown has evolved to marking a make index command to >makeindex and it produces a sitemap (if used on the web) or an index if used with an (all)TeX engine, that is fine, this can be adequately mapped. However, markup as is cannot compete with XML, which can represent a document more accurately even if LaTeX's environments come close. As XML is not for human consumption this issue needs thought, as it affects other areas of programming.

What happens next?

Simplistically the document with its markup is then passed onto a set of macros or another program for example in a browser which does a bit of processing, before passing it onto the next stage. With LaTeX or XML one can say we have a "Document Description Language" as compared to "Page Description Languages", such as PDF or raw TeX. Structuring documents this way is good and enables separation of presentation from structure and content.

LaTeX and others offers means to split this via the use of templates (i.e., what we call a class plus everything between the class and the \begin{document} command.

So far so good, but we got stuck on the Template Description Language that got stuck on programming concept fundamentals and algorithms and here neither LuaTeX or the LaTeX3 projects have offered anything substantially new. We also have limitations in touching the output routines which are currently over complicated.

As part of such a "Template Description Language", I attempted to have an ."ini" file where one would have specified settings for a template in a similar fashion to setting up an Apache or php settings file. Here LuaTeX was very good for parsing such a file (See Parsing files through LuaTeX).

Programming Constructs and Data Structures

Although I believe that (all)TeX by now has provided an adequate number of of data structures and constructs, either through the kernels or through packages, the work done by both the LuaTeX Team and LaTeX3 projects can ease the pain. Personally I would like to stay as far as possible within one programming paradigm and would prefer to stay with LaTeX3 even given it's alien syntax. On lower levels, such as font handling, encoding schemes etc. LuaTeX can be more useful.

So where to?

My general gripe with anything TeX, is the lack of agile development and a natural evolution towards goals, compensated to a great extend by the community and the rate of introduction of new packages. By natural evolution, I mean responding to user needs and providing solutions. The Teams need to address all level of users the author, template developer and programmer and so far somehow we got stuck on the user that is predominantly a programmer. It's fine with me as I tend to like puzzle solving;)

• I don't agree with you that LuaTeX does not add something substantially new. While somewhat hidden behind all the new \commands, you have a completely new access to the typesetting engine. See my sketch in the LuaTeX wiki. Or take a look at Paul Isambert's great Lua packages, for example the interpreter package (described in TUGBoat Volume 32 (2011), No. 3). This is so different from what we know today about TeX. – topskip Feb 21 '12 at 10:07
• @PatrickGundlach Maybe I did not express myself well, I do think that both Teams are adding value. However, they are solving (so far) the peripheral problems. The hard problems are still on the top. Marking a document with complex layouts and floats with a lot of buts and ifs. How to access and design output routines etc. – Yiannis Lazarides Feb 21 '12 at 10:43
• LuaTeX provides a radical new way of generating documents with TeX. It only provides the basics that must be filled in. The next question, wich is not related to TeX at all, is how to markup a document to specify typographical needs. If there is an answer, we can build such a system on top of this new TeX approach. – topskip Feb 21 '12 at 11:14
• @PatrickGundlach One can look at the web (where the user base is huge) and try and distill the lessons learned; Part of the lessons so far - let the users decide (i.e., let natural evolution take place) and to achieve this issue often, as in agile development. Radical changes work only if they disrupt a market and displace it otherwise they don't get adopted. What is the best mark-up? For computer consumption so far the winner is XML for human consumption still to be determined:) – Yiannis Lazarides Feb 21 '12 at 11:34
• A very mathematical perspective :) – Ryan Reich Feb 21 '12 at 16:18

If I had my way, I would have loved a completely new system, with a well-thought out author approach, template designer approach, and programmer approach. Importantly, any new system must be endorsed by everyone as the way forward, and the new system must be able to do anything that users can do with LaTeX today. that is, it needs access to all the typesetting power and features of existing LaTeX and CTAN.

in its first incarnation, it should have compiled into LaTeX itself. We have enough processing power nowadays to live with such an extra-stage implementation. Eventually, as decades go by, the guts would be replaced and the TeX internals itself would fade into history, replaced by a more direct typesetting engine.

Alas, it wasn't to be...

• Look for "second system symdrome" in software engineering... many times people suggested redoing everything "because now we know how to do it right", and said second try turned out dismal (witness OSI after TCP/IP). – vonbrand Feb 11 '13 at 4:37
• yes, but if we could deprecate old stuff that we learned was bad, it could become better. no one seems in charge here... – ivo Welch Feb 12 '14 at 3:54