7

First some background information.

The old distributions of TeX included two programs, initex and virtex. Initex could create format files, and virtex could load them. These older versions of TeX (and LaTeX) recommended a special installation procedure to speed up the loading of format files. It included the following approximate steps which varied depending on system:

  • initex plain \dump to load plain.tex macros, create plain.fmt and exit program
  • virtex &plain to load plain.fmt and wait for more input
  • system command to create a core dump of program, ^C, control-\ or similar
  • system command to produce an executable from the core dump, undump or similar

Now the resulting executable could be run with the plain format preloaded.

My questions concern the last two steps above:

  1. How common was it for TeX systems to use this feature during the early eighties?

  2. How much faster was it to load the format files plain.fmt and latex.fmt this way?

  3. When and why did this procedure of creating a new executable from a core dump stop being used?

  4. How did Knuth feel about recommending such a "hackish" procedure of installation?

I understand that question four might only be answerable by one person but maybe someone who was around at that time knows.

Sidenote: Early Emacs versions also employed this method and even included code to create an executable from a core image. Perl also had the option -u to create a core dump with the same purpose of speeding up the loading time.

  • 1
    You do know that we still use format files? – Joseph Wright Feb 27 '18 at 19:13
  • 1
    can't answer absolutely, but i may have old statistics somewhere. it really was faster to use a preloaded executable (as i remember, virtex was used for a particular version only until the format was solid, then an executable was made.) i don't remember when the switchover (3) came about, but even now, format files are used, particularly (i think) for luatex. re 4, i don't think at the time that knuth felt this was particularly hackish. remember, he had only a total of 1 meg of memory available, so every shortcut was necessary. – barbara beeton Feb 27 '18 at 19:18
  • I still use formats for each and every preamble I setup for everything which is beyond the scope of quick testing and answering on TeX.SX. But I don't create binaries from those, just load the *.fmt into pdflatex. (The "still" might be interpreted wrong in this context, "still" doesn't mean that I used this method in the 80s, when I wasn't even born) – Skillmon Feb 27 '18 at 19:40
  • 2
    Note that you can still compile tex this way see the --enable-auto-core option to the web2c configure script, tex.loria.fr/texlive-htmldoc/web2c/web2c_3.html#SEC15 – David Carlisle Feb 27 '18 at 22:30
  • @DavidCarlisle Interesting, I wonder when that feature was last used. – nadder Feb 27 '18 at 22:48
9

TeX was written at a time when programs operated under harsher constraints of memory (and speed), so it had to resort to many tricks and to things that are now oddities but were common at the time. Three are relevant to this question (INITEX, fmt files, core dumps); more on each of them below (though I realize the question is mainly about the last one).

These are mentioned in section 1331 (part 51: The main program) of the TeX program, as quoted in David's answer. (There's also a talk by Knuth about just these things. Over 3 days in July 1982 he gave 12 lectures addressed (I think) to (what we would call) system administrators at other universities and such places, who would install the new “portable” TeX program on their systems. This is the last of them: The Internal Details of TeX82 - Session 12. Watching those videos may answer some of the questions.)


First, the source code of TeX has some compile-time variants (like #ifdef ... #endif in C, implemented as init ... tini in the TeX program), which if turned on, result in a separate program INITEX (which has the additional ability to initialize certain internal data structures, compute hyphenation patterns, dump format files, etc). The reason for having INITEX be a separate program is that it needs extra memory to do its work (e.g. computing hyphenation patterns requires allocating memory for a trie), which leaves less memory for typesetting.

When compiled without these flags we would get a separate program, called the production version of TeX, which may either be VIRTEX or TEX (more about this later). (Today with contemporary TeX distributions e.g. TeX Live and MiKTeX, there are no longer two separate programs compiled and distributed; instead we have a single program that will behave like INITEX if given the -ini flag.)


Second format files. This is closely related to the first, as the format files are dumped by INITEX. For example, after starting INITEX (or tex -ini today) we could do \input plain and then \dump. What this does is dump out a lot of TeX's program state to a .fmt file, so that the production TeX program can simply load the format file instead of having to process plain.tex (or as it was called then, BASIC.TEX: incidentally this is why in The TeXbook the documentation of the plain format is in Appendix B: Basic Control Sequences).

In the video linked above, somewhere around the 13-minute mark, he mentions some numbers that give an idea of the order of magnitude of time saved: he says that opening and reading about a dozen short .tfm files (as INITEX or tex -ini does with \input plain) took over 15 seconds, so it was faster to read a single format file.

screenshot from §1331


Third, core dumps / undump. As I understand it, the idea was that your program's memory could be all dumped out to a file, and that you could “undump”: when your program was started all its memory would be initialized as during the dump, and the program would start at the beginning.

There is an explanation and history of this feature here, in a comment by David R. Fuchs, who (see interview: 1, 2) was Knuth's “right-hand man” when writing TeX (the current version of TeX, i.e. TeX82):

Executable program files were not much more than memory images; to run a program, the OS pretty much just mapped the executable image into your address space and jumped to the start. But when the program stopped, your entire state was still there, sitting in your address space […]

The OS also had a built-in command to allow you to SAVE the current memory image back into a new executable file. There wasn't much to this command, either, since executables weren't much more than a memory image to begin with. So, the equivalent of dump/undump was really just built into the OS, and wasn't considered any big deal or super-special feature. Of course, all language runtimes knew all about this, so they were always written to understand as a matter of course that they had to be able to deal with it properly. It pretty much came naturally if you were used to that environment, and wasn't a burden.

Thus, when TeX (and I presume the various Lisp and Emacs and etc. that were birthed on these machines) were designed, it was completely expected that they'd work this way. Cycles were expensive, as was IO; so in TeX's case, for example, it took many seconds to read in the basic macro package and standard set of font metric files and to preprocess the hyphenation patterns into their data structure. By doing a SAVE of the resulting preloaded executable once during installation, everyone then saved these many seconds each time they ran TeX.

You can read more (about other systems/programs) in the comment, and in the two discussion threads in which this came up: one on the Emacs dumper (2016), and on making the Atom editor about 50% faster (2017).

Also, in the video, Knuth says this sort of system would be available at most places, or at least the lucky ones. (13:57: “on most—anyway, on lar—on lucky systems, let's say”.) (If your system didn't have it, you could still use the format file feature: you'd start VIRTEX and load the plain format with &plain, instead of starting INITEX and \input plain.)

So the dump/undump system is not really as arcane as it may sound. And I think you have partly answered your question yourself to some extent, by pointing out that Emacs and Perl had/have similar features.


Fourth, (ha!) is the Pascal coding trick that makes the dump/undump behaviour possible. This from the TeX program:

VIRTEX not instantaneously and dirty trick pays off


Some analysis (and speculation)

Of the three tricks, the first one (INITEX as a separate program, with different memory characteristics) was just inevitable, to fit within the memory constraints. But the other two (fmt files and dump/undump) seem rather similar (for example the verb “dump” is used for both). Do we need both?

My suspicion is that the fmt file feature was added to TeX because the dump/undump feature was not available everywhere, that it's basically an implementation within TeX of what was in most (but not all) places an OS feature. So the fmt file feature was necessary, as it was the only thing guaranteed to work everywhere.

Could TeX have just the format file feature and not use dump/undump? On certain systems it was forced to, and evidently this was acceptable. On other systems (“the best implementations […] have a format file pre-loaded”, “On systems that allow such preloading…”) using dump/undump was natural so there's no reason not to. While loading a format did save a lot of time (e.g. that of opening lots of TFM files for font information), there was still work that needed to be done (“The VIRTEX program cannot read a format file instantaneously, of course…”):

  • Sanity-checking of various compile-time constants for meaningful values,
  • Initializing lots of variables to correct values (look at section 21 and over 30 “see also” sections mentioned),
  • Open and load the format file (then close it): this is the bulk of the time, and involves undumping a lot of things (section 455): the string pool, the entire dynamic memory (TeX “does nearly all of its own memory allocation”—this is basically a giant array called mem), various tables, etc. Although simply writing the final value of all of these would be faster than doing it one step at a time while processing plain.tex (opening files, defining macros one-by-one instead versus simply writing out the final hash table), it would still a fair bit of time.

So the dump/undump trick is worth doing if available. There are a few problems though:

  • Even if given the right memory, how would the Pascal program know (if started from the beginning) whether it has already read a format file (i.e. is is being started from such a dump) or not?

  • (Mentioned around 16:40) The state of the program at the time of the dump file is actually not exactly right: it has a timestamp of when you started VIRTEX to core-dump, not the time of your job, and it has the wrong \jobname. (You can try this yourself with a modern distribution: if you start with tex -ini (the closest we can get to VIRTEX), then do \input plain and then your doc (hello world \bye or whatever), then you'll get output on plain.dvi.) So we need to re-initialize some of these.

This is where the “dirty trick” comes in: by accessing a global variable that would have initialized to a specific value in the case of the core dump, but uninitialized if started from scratch, we can distinguish the two cases, and (if resuming from a core dump) initialize only those things that need initializing.

This is IMO why DEK says the trick “pays off handsomely” (making this dump/undump possible at all, and saving some time), even though it's a trick that only helps on some systems.


To answer the actual questions asked about dump/undump:

  1. This feature was common: going by the comment of DRF and the video, it was just the usual/sensible way to start (all) programs on many systems (in use at “big CS departments”). But DEK knew this feature didn't exist on all systems.

  2. It was several (20+) seconds faster to load BASIC.FMT (with VIRTEX) instead of \input BASIC (with INITEX). Not sure how much additional time saving came from the dump/undumping (starting TEX directly), but somehow I suspect it wouldn't have been more than this.

  3. The feature stopped being used when most people moved to other Operating Systems that didn't natively support dump/undump, and it needed special programs:

    But when TeX was ported over to Unix (and then Linux), it came as a bit of a surprise that the model was different, and that there was no convenient, predefined way to get this functionality, and that the runtimes weren't typically set up to make it easy to do. The undump stuff was created to deal with it, but it was never pretty, since it was bolted on.

    (Similar thing with unexec in Emacs apparently.)

  4. Based on the cues we have, we can guess Knuth wouldn't have considered it hackish (this was just how all programs worked), and would have been happy to have found a way to make TeX even faster — see David Carlisle's answer. :-)

  • But your theory that the dump/undump procedure was not crucial sort of contradicts Knuth's handsome pay off comment posted by David Carlisle. And I don't think Knuth would have put in that hack with reading of uninitialized variable unless it was really essential. – nadder Feb 28 '18 at 3:33
  • @nadder I have rewritten based on what I (think I) understand now. :-) – ShreevatsaR Feb 28 '18 at 10:42
  • That's of course very useful information but isn't it slightly off topic to write so much about things not directly related to the core dump feature? – nadder Feb 28 '18 at 14:23
  • @nadder Yes, I guess this was one of those “sorry for the long answer; I didn't have time to write a shorter one” situations :-) Actually, I was writing about all 4 steps in the installation procedures, before noticing the edit to the question clarifying that it's only about the core dumps… anyway I guess the answer is still mostly about undump, because on my computer this answer is about four screenfuls(!) long, of which only the first one is about the the other two things. Maybe I can add a header pointing to the summary at the end of the answer. – ShreevatsaR Feb 28 '18 at 18:46
  • "fmt file feature was added to TeX because the dump/undump feature was not available everywhere" - no, fmt feature is always necessary, because you have to load format to VIRTEX before doing dump/undump on it – Igor Liferenko Oct 10 '18 at 3:42
9

In answer to your question 4, he thought it "paid off handsomely"...

tex.web says:

The \.{VIRTEX} program cannot read a format file instantaneously, of course;
the best implementations therefore allow for production versions of \TeX\ that
not only avoid the loading routine for \PASCAL\ object code, they also have
a format file pre-loaded. This is impossible to do if we stick to standard
\PASCAL; but there is a simple way to fool many systems into avoiding the
initialization, as follows:\quad(1)~We declare a global integer variable
called |ready_already|. The probability is negligible that this
variable holds any particular value like 314159 when \.{VIRTEX} is first
loaded.\quad(2)~After we have read in a format file and initialized
everything, we set |ready_already:=314159|.\quad(3)~Soon \.{VIRTEX}
will print `\.*', waiting for more input; and at this point we
interrupt the program and save its core image in some form that the
operating system can reload speedily.\quad(4)~When that core image is
activated, the program starts again at the beginning; but now
|ready_already=314159| and all the other global variables have
their initial values too. The former chastity has vanished!

In other words, if we allow ourselves to test the condition
|ready_already=314159|, before |ready_already| has been
assigned a value, we can avoid the lengthy initialization. Dirty tricks
rarely pay off so handsomely.
@^dirty \PASCAL@>
@^system dependencies@>
1

From the answers and comments given so far I will try to make a summary.

  1. No one knows but it seems it was fairly common.
  2. One report indicates that it could save on average 30 seconds. See TUGboat Vol 4 No 1
  3. No one knows precisely but it coincided with TeX moving to newer operating systems which made it less convenient to do so. Late eighties, early nineties perhaps.
  4. Knuth might not have thought the actual embedding of the format in the executable a hack but he certainly considered the Pascal code he wrote to take advantage of it a hack, even calling it a dirty trick.
  • A couple of minor comments: • I think LaTeX was not yet common when TeX was being (re)written, but there was AMS-TeX and (it appears) Fácil TeX. • We can probably benchmark things even now… • The space taken by the “dirty trick” in the program itself is just one variable ready_already of type integer. • The space taken on disk would probably not have seemed a lot of extra disk space, because loading a program from core seemed the standard method at most CS departments (per DRF) — per his comments, executables were memory images basically. – ShreevatsaR Feb 28 '18 at 18:58
  • Regarding portability, I think the idea was still that a sysadmin would have to make some changes (§1332 mentions the trick should be deleted if the Pascal runtime system doesn't like it; the code has many such places, indexed under “system dependencies”). It was just a huge improvement over the earlier practice of porting by rewriting an entire program from scratch (see §2 & TUGboat “site reports”). So Knuth put whatever would be available at most (not necessarily all) systems in the program itself, and whatever would be available at very few systems into the system-dependent change files. – ShreevatsaR Feb 28 '18 at 19:09
  • 1
    In this issue of TUGboat page 20, Pavel Curtis and Howard Trickey writes For regular use, it appears necessary to have a version of the program which has the PLAIN macros preloaded. – nadder Mar 1 '18 at 2:04
  • Oh great find! And that one has an answer (for that system) at the end of the paragraph: This saves an average of 30 seconds or so of start-up time over the non-preloaded version. And going by the numbers in the next paragraph (44 seconds for a 6-page document), perhaps for a single page it may be something like the difference between getting it typeset in 8 seconds versus 38 seconds. The 30-second additional startup seems ok for batch use, but annoying for interactive use. – ShreevatsaR Mar 1 '18 at 2:14
  • Yeah I was going to type that in as well but accidentally hit return and my message was cut short, and with an ugly code quote as well. Anyway, it sounds extreme, 30 seconds more to read in a binary packed file and do a little bit of initialization. I'm guessing the pascal runtime must have been very poorly written. – nadder Mar 1 '18 at 2:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.