# Override TikZ externalize “up-to-date” flag

I am currently using the TikZ external package in order to avoid recompiling all figures in my thesis. There are more than 50 figures, relying on large data, and a full build without externalizing takes a few minutes to run. Just for some background, I use Latexmk which calls xelatex for the build.

Now my question is, per the documentation, a file is considered "up-to-date" if:

The up-to-date check is simple: if the file does not exist, it is not up-to-date. Furthermore, if one of the force remake or remake next keys is true, the figure is not up-to-date. In all other case, the file is considered to be up-to-date.

I'd like to change the requirements for a file to be up-to-date. More specifically, given the following MWE

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize
\tikzsetexternalprefix{figures/cache/}
\begin{document}
\tikzsetnextfilename{circle}
\begin{tikzpicture}
\end{tikzpicture}
\end{document}


The current behavior is to not remake the figure if the file figures/cache/circle.pdf exists. If I make a change to the tikz source, say for example \draw (0,0) circle [radius=2];, the figure will not be rebuilt.

Right now, I add \tikzset{external/remake next} before the figure, run latexmk, wait for the externalizing to start and then rapidly comment out the force remake (if I don't, the figure is rebuilt at each run, which gives huge build times).

Is there anyway to tell tikz to rebuild only if there was an edit to the source of the figure ?

[As a bonus, this should also work when using a custom external/system call, such as converting from pdf to png]

-
You say "if I don't (comment out the force remake), the figure is rebuilt at each run, which gives huge build times". How many times does latexmk run the compiler? Even if it was two or three times, this shouldn't have such a huge impact on the overall compile time since there's only one figure to be remade. –  Jake May 14 '12 at 7:13
pdfLaTeX is actually able to check for file modification date. I'm using this with standalones build feature which is an alternative to external. –  Martin Scharrer May 14 '12 at 9:08
@Jake from 3 to 4 times, the document is quite complex. The thing is, though, I like to tweak my figures, compile, tweak again, etc. The build time for each figure is ~30 seconds, which is long enough. I'd really prefer to wait less than 2 minutes between each run. Martin Scharrer: Unfortunately, I can't use pdfLaTeX, I make extensive use of some of xelatex's features. –  Adrien May 14 '12 at 12:55
@Martin modification date would not be good enough because unlike standalone, picture environments are often part of a larger TeX file (which changes frequently). I will think about checksums of the environment content; there have been enough feature requests about this already. –  Christian Feuersänger May 16 '12 at 20:08
@ChristianFeuersänger This would be awesome, although there might be a small inconvenience using a checksum of the content of the environment: if a macro defined outside changes, there would be no way of knowing that. But if it is clearly specified in the documentation that the checksum relies only on the content it should fly. Another idea would be to make the whole content of a tikzpicture environment available as a macro inside external call, this way it could be diffed against a dump of the previous version. Thoughts ? –  Adrien May 16 '12 at 20:41

As of May 17, 2012, the external lib comes with support for both checksum checks and diff-based checks (CVS version at the time of this writing).

With this version, the picture's content is compared against the state of the most recent successfull image externalization. Changes in the picture's content will automatically cause a remake for both mode=list and make and mode=convert with system call (the latter one is the system default).

The initial setup will use MD5 checksums. If MD5 checksums are unavailable, it will automatically fall back to diff-based checks.

Note that changes in the document's preamble still need to be resolved manually (by remaking all affected pictures).

Thanks for the (numerous) feature requests in this area.

-
Please provide a link to the cvs-version. –  Marco Daniel May 17 '12 at 15:53
CVS can be checked out without login from sourceforge.net/scm/?type=cvs&group_id=142562 . I fear that texample.net is no longer providing ready builds... (?) –  Christian Feuersänger May 17 '12 at 16:02
I hope @stefankottwitz (the new maintainer of the site) will add a ready build. –  Marco Daniel May 17 '12 at 16:11
@MarcoDaniel I added a build of today: texample.net/tikz/builds/2012/05/18/a –  Stefan Kottwitz May 18 '12 at 16:57
@drozzy CVS is a version control system to synchronize efforts between developers. TikZ uses it. Before someone starts research: CVS is not state-of-the-art. –  Christian Feuersänger May 18 '12 at 18:53

(Edit: This solution follows the advice of Martin Scharrer. It is not for OP since he uses XeLaTeX. But it will suit everyone who uses pdfLaTeX or LuaLaTeX.)

Place your tikzpicture in an external file (for example, image.tex) and use filemod package (from Martin Scharrer) to compare modification dates.

Here is the external tikzpicture file (image.tex)

\begin{tikzpicture}
\end{tikzpicture}


And the main document (main.tex):

\documentclass{article}

\usepackage{filemod}
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize[prefix=ext-tikz/]

\newcommand\autoupdateexternfigure[1]{
\tikzsetnextfilename{#1}
\filemodCmp{#1.tex}{ext-tikz/#1.log}%
{\tikzset{external/force remake=true}\input{#1}}
{\input{#1}}
}

\begin{document}
\autoupdateexternfigure{image}
\end{document}


To compile main document:

pdflatex -shell-escape main

-
Unfortunately, this approach has two drawbacks: first it wont work in my case because, as you said, I use XeTeX. And then there is the issue of duplication, for example I use the xcolor package, and define a few global colors which I use both inside the document and the figures, and that I don't really want to externalize… But I'm sure it'll help others, thanks for posting. –  Adrien May 17 '12 at 0:29
@Adrien Global colors used by your figures are like global lengths or global macros or packages: if they change, the only solution is to force remake all external figures. TeX can't determine automagically all dependencies. –  Paul Gaborit May 17 '12 at 7:51

Here is an overview over a couple of builtin methods to simplify up-to-date checks:

The mode=list and make is best when it comes to big projects and it has most support for up-to-date checks. Consider using make (on linux: builtin; on windows: more complicated; available as part of the cygwin bundle).

Combine it with a Makefile like

all: pgfplots_talk_FTUG_2012.pdf

include pgfplots_talk_FTUG_2012.makefile

pgfplots_talk_FTUG_2012.makefile:
pdflatex -shell-escape pgfplots_talk_FTUG_2012

pgfplots_talk_FTUG_2012.pdf: allimages FORCE
pdflatex -shell-escape pgfplots_talk_FTUG_2012

clean:
rm -f *.vrb *.toc *.nav *.out *.log *.aux *.snm *.pdf *.dep *.dpth *.djs *.figlist *.makefile tex3.tmp

FORCE:


in order to get a simple main Makefile.

PRO:

• running the main Makefile with make -j 4 will employ 4 concurrent processes at once - resulting in a speed-up of 4 for the image externalization.

• you can rely on \tikzpicturedependsonfile{<filename>} to add file dependencies. This feature is currently only available for mode=list and make. Then, whenever <filename> changes, the associated image will be rebuilt. pgfplots uses this automatically in conjunction with \addplot table and \addplot file. A possibility would be to use a separate .tex file for every picture and add \tikzpicturedependsonfile accordingly.

• this mode automatically supports \label and \ref

• no need for shell-escape

CON:

• For some people, the four characters m, a, k, e in a row cause panic attacks. Let us know if you want assistance.

• it allocates more \write registers.

Tips independent of mode = list and make:

• consider using \tikzsetfigurename{subsubsection_} (see examples in the manual). This allows to remake all figures of a complete section simply by providing the common prefix.

• use \pgfkeys{/pgf/images/include external/.code={\href{file:#1}{\pgfimage{#1}}}} in your preamble. This generates a hyperlink for every external graphics - clicking onto the hyperlink in any decent pdf viewer will open the external .pdf graphics. In addition, viewers like xpdf show the filename as tooltip/status bar. This information can be used to delete only this file from disk and to rerun the externalization. If you follow this tip, you should not forget to disable the externalization (or the key) before you finally publish the pdf (look into the pgfplots manual - occasionally, I forgot to reactivate it ;-) )

-

As the questioner uses Latexmk, I wanted to provide a possible configuration for .latexmkrc:

add_cus_dep('tikz', 'pdf', 0, 'maketikz');

sub maketikz {
system("pdflatex -shell-escape -halt-on-error -interaction=batchmode -jobname '$_[0]' '\\def\\tikzexternalrealjob{$rootfile_name}\\input{$rootfile_name}'");  } Please note, that there is no space between -interaction and =batchmode, I just couldn't fix the markup (if anyone can: please edit and remove this sentence). This configuration expects that the tikzpicture environments are located in separate files with the ending tikz. Other file endings are also possible by adapting the first argument of add_cus_dep. The command is mainly copied from the makefile generated by the external library. As the name of the main file is needed, Latexmk's variable$rootfile_name is used, as suggested by latexmk's author John Collins. He consideres documenting this variable.

This has been tested with:

\usepgfplotslibrary{external}
\tikzexternalize[mode=list and make]


It should be mentioned, that this is mainly useful if only one tikzpicture is changed at a time, i.e. during the normal writing workflow. If all figures have to be (re-)generated, the makefile should be used, as it allows for parallelizing.

-