# How to exploit externalized images runtime-wise?

Thanks to the valiant efforts of @zeroth, I got TikZ' externalization running with todonotes, TikZ images pieced together by environments and macros and my homebrew compilation script. In particular, I do not use -shell-escape but run the external jobs "by hand" (better for output stream collection).

In my preamble, I have

\usepackage{tikz, pgfplots}
\usetikzlibrary{external}
\tikzexternalize
\tikzset{external/mode=list and make}%
\tikzset{external/export=false}%


and external/export is turned on for individual images. There are no other explicit (global) TikZ settings, not counting style definitions.

The results have been sobering, though: build-time about doubled!

Apparently, translating the images on their own is way more expensive than just translating them as part of the main file. Of course, pdflatex calls two to n (five, in my case) are way faster but that does not seem to offset the overhead. The document this was observed with contains about forty images, the majority plots (pgfplots + gnuplot + data files).

I could live with that if only the first build took that long. However, tikz/external does apparently not check whether an image has changed since the last build; it always creates a complete .figlist and I (that is my conservative script) have to rebuild all of them.

Of course I could check for each figure wether the fitting PDF is there and not rebuild if it is. However, that would cause changes to the image to be ignored until rebuild is forced somehow.

I can see no helpful auxiliary outputs that I (my script) could use to check whether an image has changed. Therefore my question:

Is there a way

• force TikZ to check for still valid externalization results or
• to detect from the outside whether it is necessary to rerun TikZ externalization jobs?
-
See my post in the chat room. Use the makefile approach which will determine whether updates are necessary for the images based on the external files that are used. As for compile time, it is simply because the externalisation compiles the entire document. And only removes a few commands which are optimized away. –  zeroth Apr 8 '12 at 19:08
@zeroth: 1) As explained, the make file won't be able to catch everything (especially changes hidden inside the .tex files) and is project-specific, what I want to avoid. But thanks for the hint. 2) If I can believe the log files of the external calls, all TikZ images are optimised away and those cause the brunt of runtime. I am open for suggestions to reduce the runtime of external calls further. –  Raphael Apr 8 '12 at 19:16
Changes in TeX files is really hard to catch, unless you have each figure in a seperate file and check timestamps against those. In the latter you can easily check for PDF vs. .tex. As for the runtime check which commands in your document takes the longest time and add them to the optimization scheme as explained in the manual. –  zeroth Apr 8 '12 at 19:18
At the moment the externalization is not built to catch what you request. –  zeroth Apr 8 '12 at 19:19
@zeroth: You can catch changes in external files in pdfTeX with the help of \pdffilemoddate and \pdfmdfivesum. Heiko Oberdieck uses them in his packages. –  Martin Schröder Apr 9 '12 at 13:49

I fear that the external lib of TikZ addresses a slightly different use-case than what I understand here.

The external lib has been built with the following assumptions/requirements in mind:

1. allow image externalization without changing the original TeX document (except for non-intrusive changes to the preamble)

2. once an image has been externalized, it is assumed to be static and the external image can be reused lots of times.

Consequently, time-savings can be expected if and only if (2.) is satisfied. If (2.) is satisfied, the external lib will reduce time.

However, as I understand from your comments, you are recomputing each figure for every build. In such a scenario, the external lib "suffers" from its requirement (1.) and a "missing feature" of pdftex: in order to create one external image, it has to process the main .tex file (and throw away everything that does not belong to the picture).

While "throw away everything that does not belong to the picture" allows a couple of optimizations which reduce compilation time significantly (like throwing away all non-enclosed tikzpictures and all non-enclosed \includegraphics-statements), it is expected behavior that the total runtime to create each picture + the main .tex file using external pictures increases compared to processing the main .tex file without externalization at all.

A factor of two is unfortunate, but might happen for a huge document.

Here are my suggestions to overcome the problem:

Solution 1 Consider using the standalone-package instead of the TikZ external lib. It also causes images to be externalized, but it does not have requirement (1.) and therefor leads to a considerably smaller overhead. It might be closer to what you do currently.

Solution 2 Consider changing your Up-to-Date check. Is it really necessary to recreate each external graphics every time you run the build? Why is that necessary? Can the criterion be relaxed at the cost of reliability?

In other words: try to satisfy assumption (2.): once the majority of images does is static, you will experience significant time-savings.

In order to get a reliable solution, you should still rerun the complete process before you finally publish your .pdf. Depending on the sort of pictures, you may even want to disable the external library completely for the final runs (if you dynamic content in your images like the clickable library of pgfplots).

Solution 3 You can accept that the external lib does not fit your current use-case and work-around its weaknesses -- by customizing its optimization feature (i.e. to reduce the time to determine which portions of the main .tex file can be skipped safely).

-
I forgot to mention: if you have a makefile, you can probably externalize multiples files at ones if you have a multicore system. GNU makefiles support -j 4 (for 4 threads). –  Christian Feuersänger Apr 12 '12 at 8:34
I ended up assuming 2. and building only missing figures anew. If we drop 2., we get into trouble; without assumptions on the document there is no way to figure out whether a figure changed (even disregarding references). Hopefully, external will be extended to make those checks before writing \$job.figlist. –  Raphael May 14 '12 at 15:57
@Raphael If you have an idea about realistic expectations of such an builtin up-to-date check, I would be interested. I guess the checksums of the tikzpicture environment is the best we could get - and it suggests a safety which is not given because it does not (cannot) cover changes in global styles. –  Christian Feuersänger May 15 '12 at 19:58
That, and there may be included files (\input, but also data files from pgfplots). I don't know how TikZ works; I assumed it had some kind of "maximally unfolded" version in hand at some point. I guess the messiness of TeX makes it hard for us. –  Raphael May 15 '12 at 20:35
Apparently, there has since been added an option external/up to date check which uses some basic heuristics for determining whether an image should be rebuilt. –  Raphael Nov 26 '12 at 20:08