Custom format file: How to automate the generation of the “precompiled preamble” whenever one of the files that are used there are changed

Question

I would like to speed up compilation of my LaTeX documents using the precompiled preamble approach, and I was wondering whether any of the Tools for automating document compilation have support for this.

The main idea is:

• split the document into a "static" and a "dynamic" part, preamble.tex and main.tex

• create a custom format file main.fmt from the static part using

    pdflatex -ini -jobname=main "&pdflatex preamble.tex\dump"
  

• compile your document using

    pdflatex -fmt main main
  

As a consequence, only the contents of main.tex are processed on "regular" compilation, thus accelerating the build. However, if preamble.tex or any of the files used there changes, the format file will have to be rebuilt. I would like to automate this: Rebuild main.fmt if and only if necessary. Of course, main.tex would also be recompiled if the static part (=the format file) has changed and/or if the contents or includes of the dynamic part have changed.

I have tried latexmk and scons, both excellent tools otherwise, but no luck so far.

EDIT: Philipp's answer reminded me that portability between platforms cannot be taken for granted. I'm running Linux, but I'd be most interested in fully portable solutions.

MWE

preamble.tex:

\documentclass{article}
\usepackage{lipsum}

main.tex:

\begin{document}
  \lipsum
\end{document}

Answer 1

See my solution using latexmk at Latexmk can't see a dependency on a .fmt format file. It now tracks changes in the files used to make the format file.

Answer 2

The following make-based approach provides the automatic re-generation of the precompiled preamble if any of its dependents (transitive closure) have changed.

It is based on -recorder option of pdflatex that, when compiling a file generates a list of all INPUT and OUTPUT files that had been accessed during the process. This list is then transformed by some sed script to extract a list of make-compatible dependencies. Using this list, make automatically rebuild the preamble if one of its input files have changed.

Minimal Working Example

First, the source files needed to test the approach (MWE)

preamble.tex:

\documentclass{article}
\usepackage{lipsum}
\usepackage{tikz}

\input{figure}

Note that, besides some packages, preamble.tex also \input's a local file figure.tex:

\newcommand{\myfig}{%
\tikz \draw[thick,rounded corners=8pt] 
(0,0) -- (0,2) -- (1,3.25) -- (2,2) -- (2,0) -- (0,2) -- (2,2) -- (0,0) -- (2,0);
}

In the example this serves a local dependency for the precompiled preamble. Hence, editing figure.tex or a simple touch figure.tex should cause the precompiled preamble to be re-generated.

Finally, we have our main.tex

\begin{document}
  \myfig
  \lipsum
\end{document}

So here comes the Makefile that puts it all together. Just do a make and it will generate the precompiled preamble main.fmt and then main.pdf. Subsequent changes to main.tex will not trigger a regeneration of main.fmt. If, however, you touch preamble.tex or figure.tex, this will be detected and main.fmt will be re-generated:

#####################################################################
#
# Generate depencendcies for precompiled preamble 
#
#####################################################################

# standard options passed to sed: include only lines beginning with INPUT, 
# remove the INPUT as well as trailing ./ from paths
CREATE_DEPS_OPTIONS = -e '/^INPUT /!d' -e 's/^INPUT //' -e 's/^\.//'  

# in order to not check large parts of the local/global texmf-tree we exclude all paths that start with: 
# / or texmf/ 
DEPS_EXCLUDE ?= ^/ ^texmf/ 
CREATE_DEPS_OPTIONS += $(foreach d,$(DEPS_EXCLUDE), -e '/$(subst /,\/,$(d))/d')


# This includes the extracted deps
MAIN_DEPS= $(shell touch main.fls ; \
                            cat main.fls | \
                            sed $(CREATE_DEPS_OPTIONS) | \
                            sort -u | \
                            xargs)

all: main.pdf

main.fmt : $(MAIN_DEPS)
    @echo "****************************************"
    @echo $? changed, hence recompiling precompiled preamble 
    @echo "****************************************"
    @pdflatex -recorder -ini -jobname=main "&pdflatex preamble.tex\dump"

main.pdf : main.fmt main.tex
    @pdflatex -fmt ./main main

clean :
    @rm *.log *.aux *.pdf *.fmt *.fls

Implementation Details

The command

pdflatex -recorder -ini -jobname=main "&pdflatex preamble.tex\dump"

builds the precompiled preamble main.fmt with the -recorder option. With this option, pdflatex generates a list of input and output files main.fls it accessed during the run that looks as follows:

PWD /Users/lohmann/projects/stackexchange/precompiled
INPUT /opt/local/etc/texmf/texmf.cnf
INPUT /opt/local/var/db/texmf/web2c/pdftex/pdflatex.fmt
INPUT preamble.tex
OUTPUT main.log
INPUT /opt/local/share/texmf-texlive-dist/tex/latex/base/article.cls
INPUT /opt/local/share/texmf-texlive-dist/tex/latex/base/article.cls
INPUT /opt/local/share/texmf-texlive-dist/tex/latex/base/size10.clo
INPUT /opt/local/share/texmf-texlive-dist/tex/latex/base/size10.clo
INPUT /opt/local/share/texmf-texlive-dist/tex/latex/lipsum/lipsum.sty
INPUT /opt/local/share/texmf-texlive-dist/tex/latex/lipsum/lipsum.sty
... <lots of more data>
INPUT figure.tex
INPUT figure.tex
OUTPUT main.fmt

In the definition of the MAIN_DEPS variable, the INPUT lines from this list are extracted (via sed and xargs) into a sequence of file names that can be used as make dependencies. With DEPS_EXCLUDE one can pass regular expressions of patterns that should be ignored; in the example I use this to ignore dependencies from the global texmf tree to speed up the deduction process within make. If you really want all dependencies to be checked every time, just set it to an empty string.

Answer 3

As per @Yossi's suggestion, you could use make with rules similar to:

main.pdf: main.fmt main.tex <other dependencies>
        pdflatex -fmt main main

main.fmt: preamble.tex <other dependencies>
        pdflatex -ini -jobname=main "&pdflatex preamble.tex\dump"

Of course, you can replace the command pdflatex -fmt main main to generate the document with some other command that does the job better, for example latexmk to be sure that references are resolved.

Answer 4

If you are using MS Windows then a little program called LaTeX Daemon might be interesting for you.

Answer 5

WhizzyTeX (for Emacs) does this. Normally it is used to only compile a portion of the file, but it can be made to run the entire file as well. It actually saves a format file at the start of each "slice" (portion of the code WhizzyTex is compiling e.g. section, subsection, slide).

WhizzyTeX is meant as sort of a live-update, which might be what you're going for. This may not be a solution for you (e.g. if you don't use Emacs), but it might give you some ideas.

Answer 6

Here is a solution in bash. Not portable, but easily adaptable.

#!/bin/bash

####################
# CONFIGURATION
#
PREAMBLE="preamble.tex"
MAIN="./main.tex"

####################
# JOB
#
JOBNAME=$(basename ${MAIN%.*})
FMT="$(dirname "$PREAMBLE")/$JOBNAME.fmt"

COMPILE=0
if test "$FMT" -ot "$PREAMBLE"; then
    # If the preamble file has been changed
    COMPILE=1
else
    # If one of the file called by input or include in
    # the preamble file has been changed
    for input in $(grep -oE "\\\\(input|include){.+}" "$PREAMBLE"); do
        file=$(dirname "$PREAMBLE")/${input:$(expr index "$input" "{"):-1}.tex
        if test -f "$file"; then
            if test "$FMT" -ot "$file"; then
                COMPILE=1
                break
            fi
        else
            echo "Error: file '$file' (called in '$PREAMBLE') is not reachable." >&2
            exit 1
        fi
    done
fi

# Changing working directory
ORIGDIR=$PWD
cd $(dirname "$PREAMBLE")

# If some file was modified since the last compilation, we
# have to recreate the precompiled preamble
if [ $COMPILE -gt 0 ]; then
    pdflatex -ini -jobname=$JOBNAME "&pdflatex $(basename $PREAMBLE)\dump"
fi

# Changing working directory
cd $(dirname "$MAIN")

# Each time, we run the command on the main .tex file
pdflatex -fmt $(dirname "$PREAMBLE")/$JOBNAME $JOBNAME

# Returning back
cd $ORIGDIR