Override TikZ externalize “up-to-date” flag

Question

I am currently using the TikZ external package in order to avoid recompiling all figures in my thesis. There are more than 50 figures, relying on large data, and a full build without externalizing takes a few minutes to run. Just for some background, I use Latexmk which calls xelatex for the build.

Now my question is, per the documentation, a file is considered "up-to-date" if:

The up-to-date check is simple: if the file does not exist, it is not up-to-date. Furthermore, if one of the force remake or remake next keys is true, the figure is not up-to-date. In all other case, the file is considered to be up-to-date.

I'd like to change the requirements for a file to be up-to-date. More specifically, given the following MWE

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize
\tikzsetexternalprefix{figures/cache/}
\begin{document}
\tikzsetnextfilename{circle}
\begin{tikzpicture}
  \draw (0,0) circle [radius=1];
\end{tikzpicture}
\end{document}

The current behavior is to not remake the figure if the file figures/cache/circle.pdf exists. If I make a change to the tikz source, say for example \draw (0,0) circle [radius=2];, the figure will not be rebuilt.

Right now, I add \tikzset{external/remake next} before the figure, run latexmk, wait for the externalizing to start and then rapidly comment out the force remake (if I don't, the figure is rebuilt at each run, which gives huge build times).

Is there anyway to tell tikz to rebuild only if there was an edit to the source of the figure ?

[As a bonus, this should also work when using a custom external/system call, such as converting from pdf to png]

Answer 1

As of May 17, 2012, the external lib comes with support for both checksum checks and diff-based checks (CVS version at the time of this writing).

With this version, the picture's content is compared against the state of the most recent successfull image externalization. Changes in the picture's content will automatically cause a remake for both mode=list and make and mode=convert with system call (the latter one is the system default).

The initial setup will use MD5 checksums. If MD5 checksums are unavailable, it will automatically fall back to diff-based checks.

Note that changes in the document's preamble still need to be resolved manually (by remaking all affected pictures).

Thanks for the (numerous) feature requests in this area.

Answer 2

(Edit: This solution follows the advice of Martin Scharrer. It is not for OP since he uses XeLaTeX. But it will suit everyone who uses pdfLaTeX or LuaLaTeX.)

Place your tikzpicture in an external file (for example, image.tex) and use filemod package (from Martin Scharrer) to compare modification dates.

Here is the external tikzpicture file (image.tex)

\begin{tikzpicture}
\fill[green] circle[radius=1cm];
\end{tikzpicture}

And the main document (main.tex):

\documentclass{article}

\usepackage{filemod}
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize[prefix=ext-tikz/]

\newcommand\autoupdateexternfigure[1]{
  \tikzsetnextfilename{#1}
  \filemodCmp{#1.tex}{ext-tikz/#1.log}%
  {\tikzset{external/force remake=true}\input{#1}}
  {\input{#1}}
}

\begin{document}
\autoupdateexternfigure{image}
\end{document}

To compile main document:

pdflatex -shell-escape main

Answer 3

Here is an overview over a couple of builtin methods to simplify up-to-date checks:

The mode=list and make is best when it comes to big projects and it has most support for up-to-date checks. Consider using make (on linux: builtin; on windows: more complicated; available as part of the cygwin bundle).

Combine it with a Makefile like

all: pgfplots_talk_FTUG_2012.pdf

include pgfplots_talk_FTUG_2012.makefile

pgfplots_talk_FTUG_2012.makefile:
    pdflatex -shell-escape pgfplots_talk_FTUG_2012

pgfplots_talk_FTUG_2012.pdf: allimages FORCE
    pdflatex -shell-escape pgfplots_talk_FTUG_2012

clean:
    rm -f *.vrb *.toc *.nav *.out *.log *.aux *.snm *.pdf *.dep *.dpth *.djs *.figlist *.makefile tex3.tmp

FORCE:

in order to get a simple main Makefile.

PRO:

• running the main Makefile with make -j 4 will employ 4 concurrent processes at once - resulting in a speed-up of 4 for the image externalization.

• you can rely on \tikzpicturedependsonfile{<filename>} to add file dependencies. This feature is currently only available for mode=list and make. Then, whenever <filename> changes, the associated image will be rebuilt. pgfplots uses this automatically in conjunction with \addplot table and \addplot file. A possibility would be to use a separate .tex file for every picture and add \tikzpicturedependsonfile accordingly.

• this mode automatically supports \label and \ref

• no need for shell-escape

CON:

• For some people, the four characters m, a, k, e in a row cause panic attacks. Let us know if you want assistance.

• it allocates more \write registers.

Tips independent of mode = list and make:

• consider using \tikzsetfigurename{subsubsection_} (see examples in the manual). This allows to remake all figures of a complete section simply by providing the common prefix.

• use \pgfkeys{/pgf/images/include external/.code={\href{file:#1}{\pgfimage{#1}}}} in your preamble. This generates a hyperlink for every external graphics - clicking onto the hyperlink in any decent pdf viewer will open the external .pdf graphics. In addition, viewers like xpdf show the filename as tooltip/status bar. This information can be used to delete only this file from disk and to rerun the externalization. If you follow this tip, you should not forget to disable the externalization (or the key) before you finally publish the pdf (look into the pgfplots manual - occasionally, I forgot to reactivate it ;-) )

Answer 4

As the questioner uses Latexmk, I wanted to provide a possible configuration for .latexmkrc:

add_cus_dep('tikz', 'pdf', 0, 'maketikz');

sub maketikz {
    system("pdflatex -shell-escape -halt-on-error -interaction=batchmode -jobname '$_[0]' '\\def\\tikzexternalrealjob{$rootfile_name}\\input{$rootfile_name}'");

}

Please note, that there is no space between -interaction and =batchmode, I just couldn't fix the markup (if anyone can: please edit and remove this sentence). This configuration expects that the tikzpicture environments are located in separate files with the ending tikz. Other file endings are also possible by adapting the first argument of add_cus_dep.

The command is mainly copied from the makefile generated by the external library. As the name of the main file is needed, Latexmk's variable $rootfile_name is used, as suggested by latexmk's author John Collins. He consideres documenting this variable.

This has been tested with:

\usepgfplotslibrary{external}
\tikzexternalize[mode=list and make]

It should be mentioned, that this is mainly useful if only one tikzpicture is changed at a time, i.e. during the normal writing workflow. If all figures have to be (re-)generated, the makefile should be used, as it allows for parallelizing.