5

One of my biggest problems with LaTeX is the speed that it takes to process large documents. (I typeset books with LaTeX.) I have approaches, such as breaking the book into chapters and running them independently. But the results are not satisfactory.

For example, the book I'm currently working on reports this after I run LaTeX:

Latexmk: All targets () are up-to-date
no errors
make  74.57s user 1.92s system 99% cpu 1:17.05 total

My computer has six cores! LaTeX uses one.

EDIT

Details of my system:

  • MacMini (2018) 3 Ghz 6-Core Intel Core i5 Processor, 32 GB 2667 MHz DDR4 RAM
  • 41 .tex input files, total of 11,000 lines of LaTeX source, over 100,000 words of text.
  • Moved to xelatex due to Unicode issues, but pdflatex took roughly the same period of time.
  • 69 included packages
  • Book typesets to 326 pages currently, will typeset to 500 at completion of project.
  • 72 images, most in the neighborhood of 20-50k.
  • Output logfile is 5611 lines long (!)
  • Still using BiBTeX because biber breaks, and I can't debug it. But using biblatex, at least.
  • Compiling with latexmk
  • Multiple targets in Makefile, including targets to just typeset each chapter. Typesetting a single chapter can be done in 15.97 seconds

LaTeX experts will say that every page depends on everything that comes before it because LaTeX is Turing Complete. And, of course, every page also depends on every page that comes after it, because of tempfiles. Frankly, it's amazing that LaTeX converges when typesetting!

However, there are well-known tricks that could be used to solve this problem. For example, pages could checkpoint relevant state, and then a new compile could use a multi-threaded implementation and run each page using the previous run's checkpoint, and re-run the pages if the checkpoint changes. Something similar could be used to speed even single-threaded runs: if we're compiling page 265, and the state at the beginning of page 265 is the same this run as the previous run, and no text in the document has changed between the start of page 265 and the start of page 266, then nothing on page 266 probably changed.

It seems that optimizations like these could make LaTeX dramatically faster. So why don't we see them in LaTeX2e, and will they be in LaTeX3?

23
  • 3
    It is likely for LaTeX to benefit from concurrent implementations. But for an open-source project like this, which has accumulated decades of legacy code, the motivation to realize such an implementation is low, and the difficulty is high. Splitting up the document can greatly reduce compilation time. If done correctly, it should not change the layout of the document by much. Another factor I find out is that LaTeX distributions on Windows typically run much slower than their Linux counterparts. If you are using Windows, maybe you should try pdflatex on Linux instead. – Alan Xiang Oct 10 '20 at 3:07
  • 8
    If you are writing a book do you really need to compile the whole thing each time? Just work on the chapter you are working on: that typically speeds things up a lot and compared to the time spent writing , a few seconds to remake a chapter is rarely an issue. Speed is an issue in other workflows eg if you are using tex to typeset 100s of reports or invoices from some database, saving a few milliseconds each run adds up. But in human oriented document authoring less so. – David Carlisle Oct 10 '20 at 8:19
  • 2
    Back in 1991 when I had my first PC, I was running a 32-bit version of LaTeX on a machine with 1Meg of RAM. With all the memory churn, it took about an hour to compile a single chapter of the class notes for the LaTeX class I was teaching. There was no multitasking of course, so I got a lot of recreational reading done although I had to keep an eye on the computer lest there be an error. – Don Hosek Oct 10 '20 at 19:29
  • 1
    @vy32 Sure, my github username is "shreevatsa" and I'd like to take a look. But probably one of the recommendations would be to not have multiple runs each time, and just accept out-of-date references (or an incomplete index) in draft mode... – ShreevatsaR Oct 18 '20 at 7:55
  • 2
    BTW you can run latexmk with the -time option (and optionally -silent) to see output at the end about what took how long. In this case, it seems that the bulk of the 70-odd seconds is taken by four calls to xelatex each taking about 17 seconds. (Interspersed with calls to makeindex and bibtex.) So if you don't need the index and all references each time, you can just run xelatex and save about three-fourths of the time. (The “full” version can still be run in the background while you're reading the document I guess.) – ShreevatsaR Oct 19 '20 at 4:48
18

The LaTeX project does spend a lot of time making sure that latex goes as fast as possible but none of the things you suggest are relevant to latex code; you are suggesting changes to the tex language in which latex is written.

As you can see if you look through github issues, a lot of thought goes into optimising the core expl3 programming constructs, whether it's quicker to have multiple \expandafter or \fi-delimited argument or to use \expanded or whatever in each case.

Also LaTeX releases this year have preloaded two largish packages into the format, expl3 (in February) and xparse (in October) this can make a fairly noticeable improvement in startup time as locating package files and reading the data off the filesystem can take significantly longer than processing the tex code within the file.

Note you can build a custom format pre-loading the packages you use which can also speed up start up time a lot.

The kind of check pointing that you mention is asking about the underlying tex system so not addressable within LaTeX. It is the same as other programming languages. A Web page author can avoid inefficient JavaScript in their page to make the page load faster, they can not re-write the JavaScript engine in all possible browsers in which that code may run, which is the equivalent of what you are asking here.

The actual checkpointing is hard as page breaking is asynchronous, it is feasible at forced page breaks from \clearpage which is exactly what the LaTeX \include system does, it saves the state of all LaTeX counters at that point so if on the next run you skip chapters 1-3, the page numbering is preserved and the draft document starts with chapter 4. But to do that automatically and to save more state, such as the definitions of all macros, not just the values of all counters, would require changes to the tex system not to latex.

You mention that later changes can affect earlier ones due to auxiliary files, but that is actually the easier case, just consider a long paragraph that spans over two or more pages. Adding a comma in the last line can change the line breaking of the entire paragraph so changing earlier pages without any auxiliary files being involved.

Many tex systems these days are fast enough that latex is set up to run continuously in the background as the file is edited and update the display whenever the pdf is successfully remade, if your build is slow you should look to your build system, are you including high resolution images or re-setting complicated tikz on every run? If you arrange to save these things to more occasional "full" builds you can usually get things to run at a reasonable speed.

15
  • Thank you so much for the detailed comment. The difference between what's in TeX and what's in LaTeX is always a bit confusing to me. And thank you for reminding me about the difference of \input vs. \include. Alas, I am using \include and getting a separate .aux file for each chapter in my book, but all of the .aux files seem to be re-generated with every pass. – vy32 Oct 10 '20 at 17:45
  • I should have clarified that because of labels and what not--including captions---it is quite easy for a change in a later chapter to be reflected earlier. If I have a list of tables in the beginning and add 10 tables in chapter 10, then my list of tables may suddenly take an extra page, forcing every page in the book to change. – vy32 Oct 10 '20 at 17:46
  • Also, it's true that many tex systems are fast, but even Overleaf is taking about 2 minutes to typesert the book. It's not a lot of high resolution images or running tikz. (I actually have a mode to incorporate low-resolution versions by default.) It's just a big book. – vy32 Oct 10 '20 at 17:47
  • 4
    @vy32 well I remember when 15 minutes per page was thought fast, so 2 minutes per book doesn't seem that slow but overleaf isn't necessarily the fastest compared to a local install on a local disk and as I say how often do you need to typeset the whole book? if it takes 2minutes to do a full build once or week or so how much of an issue is that? – David Carlisle Oct 10 '20 at 18:01
  • 5
    @vy32 if you are using \include then add \includeonly{chapter4} or whatever the filename is called, and it will just process the one chapter while keeping all cross references to skipped chapters. that is the whole point for \include and why latex has that command not \input. It is there specifically to address your issue. – David Carlisle Oct 10 '20 at 18:03
3

Putting this here as a bit of a provocative manifesto:

Popping back into LaTeXworld lately I've contemplating finally finishing the LaTeX book that I'd started in the late 80s/early 90s when I taught the TUG LaTeX classes.

LaTeX 2e was first released in 1994 as a transitional step to the eventual release of LaTeX 3. 26 years later, there still isn't a 1.0 release of LaTeX 3. In the interim, we've seen the rise of HTML and the web, the dominance of PDF as a format for representation of printed material (and now there is a plan to have PDF extended with "liquid mode" that allows reflowing of PDF text for smaller screens).

In the meantime, the TeX engine has been extended multiple times, the little-used TeX-XeT, some early efforts to support large Asian character sets, and we have in widish use pdfTeX, XeTeX, LuaTeX along with an assortment of abandoned engines. Worst of all, it seems that none of pdfTeX, XeTeX or LuaTeX can serve as the one TeX to rule them all, each with some limitations that can require users to switch engines depending on their needs.

As I've thought about it, the problem at its root is TeX itself. It's what would be referred to in contemporary software engineering parlance, as a tightly-coupled monolith. Worse still, it's a tightly-coupled monolith with numerous compromises baked in because of the limitations of 1970s computing hardware. It seems that the vast majority of what work has been done with LaTeX 3 has been geared towards dealing with the limitations of TeX as a programming language.

On top of that, there's been an explosion of questionable, if not outright harmful practices from the greater LaTeX community. Ideally, a document should be translated from one document class to another structurally similar class (naming-wise, the choice of "class" to name document classes is unfortunate, but understandable) should not require changing anything after the preamble, better still, nothing but the \documentclass command itself. All the appearance should be handled through the document class and packages should be employed to provide document structure enhancements or new capabilities). There are numerous violations of this. The memoir class is a mess, claiming to be a replacement for article, report and book (this reminds me of the mess that's PHP where the same data structure acts as an array and an associative array and as a consequence manages to merge the worst aspects of both in one inefficient construct) and at the same time, providing a number of bits of functionality that belong in packages rather than the document class. On the flipside, packages like geometry and fancyhdr fall into a category that LaTeX2e doesn't really define, bits of common code that would be helpful to document class writers but shouldn't really be exposed to document authors.

Given the ultimate failure of NTS and ExTeX, I'm not hopeful for a resolution to any of these issues.

Edit 20 October 2020 I've done something stupid. I'm starting a new project. It will probably never amount to anything.

7
  • Thank you for this comment, and I'm in complete agreement with you. It is surprising that the underlying TeX engine has so ossified, but on the other hand, that is what Knuth wanted for stability. I recently met a (surprisingly not-famous) old timer who has exteded TeX to allow editing on the displayed document and then the changes automatically get back-propigated to the underlying TeX document. He thought that the TeX community would not be receptive to the mods because it wouldn't automatically work with all of the myriad of LaTeX packages. – vy32 Oct 12 '20 at 4:45
  • 2
    LuaTeX (or at least LuaHBTeX) does combine more-or-less all of the improvements that pdfTeX and XeTeX offer, so in many ways it's the future engine for everyone. However, it's deliberately not 100% output compatible with pdfTeX (and thus TeX90), which is an issue for some people. And if we are talking performance, Unicode is always slower than 8-bit. – Joseph Wright Oct 12 '20 at 8:54
  • 2
    There are of course alternatives to the TeX approach that still use the ideas from TeX: most obvious would be Speedata (uses LuaTeX but from Lua other than a couple of bootstrap lines). The issue as always is not having clever ideas but getting to a critical mass. The LaTeX team have looked hard at a new format, and we've concluded that it's never going to get that critical mass, hence focussing on improving LaTeX2e. – Joseph Wright Oct 12 '20 at 8:56
  • Engine work is a whole different thing to macro programming, but you might look at HINT and JSBox as possible 'future directions' (both yet to be really tested in the wild) – Joseph Wright Oct 12 '20 at 8:57
  • 1
    Well, good luck with your new project (and I mean it sincerely); I hope you can take some lessons from the menagerie of failed (La)TeX replacements. You may also find it useful to talk to people who have tried writing alternative typesetting engines (SILE, Patoline, etc.) or look at people who have worked closely with the TeX engine code itself (developers of XeTeX, LuaTeX, jsBox, HINT… and also BaKoMa TeX, Texpad, SwiftLaTeX) . Anecdote from Doug McKenna of jsBox: he told Knuth he was writing a new TeX engine and Knuth warned him it would take 5 years of full-time work, and was exactly right. – ShreevatsaR Oct 21 '20 at 6:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.