tikzexternalize in parallel
I thought I might as well expand my answer a bit regarding the parallelization of externalizing
tikzpictures. I recommend reading the section of pgfman for more details (p.607 onwards).
Disclaimer: I'm a Linux guy; haven't tried this on other platforms. But this question suggests that it can be done, apparently.
Anyway: Let's make a very simple example document with quite a few
tikzpictures in it. Note the
mode=list and make in the call to
tikzexternalize. That's the important bit.
\tikzexternalize[prefix=tikz-,mode=list and make]
\draw (0,0) -- (1,1);
\draw (1,0) -- (1,1);
\draw (0,2) -- (1,1);
\draw (0,3) -- (1,1);
\draw (0,0) -- (2,1);
\draw (0,0) -- (5,1);
\draw (0,0) -- (2,-1);
\draw (0,0) -- (1,6);
\draw (0,1) -- (7,1);
\draw (0,-1) -- (5,1);
\draw (-1,0) -- (1,0);
or your engine of preference.
Then you will have a new file in your directory,
myfile.makefile. Execute that with
make -j<PARALLEL_JOB_COUNT> -f myfile.makefile
And that will build your
tikzpictures in parallel batches of
<PARALLEL_JOB_COUNT> For four, this would be
make -j4 -f myfile.makefile
Use your number of cores as a guideline for how many parallel jobs to run.
If all has gone well, you can then compile your document again and include all the
And all should be well.
As I mentioned in the comment to TeXnician's answer, I was able to cut down compilation time for my thesis (lots of
pgfplots) from about 90 minutes to roughly 13 on a clean directory (well, 16 minutes, actually, counting the two 90-second runs before and after generating the
tikzpictures, but still -- pretty impressive gain, methinks). Of course, more cores are better here; I have 12 on that machine.
Aside from compilation time, another thing I found hugely beneficial to this process came about in the workflow though: You can easily force a rebuild of individual tikzpictures from the commandline. If you have your individual
tikzpictures in separate files, this means you basically get a very similar workflow as if you'd used
standalone for your
tikzpictures. So, for example, if we want to build
tikz-main-figure5.pdf in this case, we can issue:
make -B -f main.makefile tikz-main-figure5.pdf
-B option forces
make to rebuild the target -- usually needed in these cases if you change your
tikzpicture without then compiling your main document again, because then the
m5dsum will not get updated and
make will think nothing has changed. You can of course also just delete the
.pdf file and recompile without the
-B switch, but this is how I usually did it during those long hours late at night when I was fiddling with my plots and trying to get them to look right. I had the TikZ pdf open in one window, and the editor with the source code in another, and a shortcut for the recompilation, and that was really quite convenient to work with.
Leaving aside the things which have already been mentioned about the compilation of a TeX document being a heavily serialized job in many ways, you would probably need to write your own TeX implementation to run on your GPU, or a wrapper which can run the x86 TeX on your GPU. Neither of these are what I'd call a trivial process, and given that in many scenarios the benefits would probably be marginal (or negative) at best, I doubt it's worth it.
Update: Benchmarks and Background
Since a compile time of 90 minutes seems as ludicrous to me as it probably does to most here, I've dug up the project again and have been fiddling around with it a bit. The current numbers are as follows:
- Building all
tikzpictures from scratch (most of which are
pgfplots), sequentially: 57 minutes
- Building the
tikzpictures with 12 jobs in parallel: 7:30 minutes
The whole shebang consists of
- 61 output pdf files for the
- 256,000 lines of CSV data, 130,000 of which is 3 columns, the rest mostly 2 columns
\addplot commands, the data for which is usually read in via
At the moment, I see two primary areas for optimization. Since they are also generally valid for speeding up the compilation of
tikzpictures, I hope this is still sufficiently on-topic to mention:
- Reducing the number of data points, and
- removing any
filter/.code commands, moving their work to the tool which generates the data in the first place.
- Yes, there is potential for optimization here.
- I'm fine with that. My point was that parallel compilation of
tikzpictures can bring significant reductions in overall compile time and is relatively easy to accomplish if you're not commandline-averse.
- Also, it's not as bad as it sounds, since I rarely have to compile all
tikzpictures from scratch -- usually I just update one thing, and then the whole document compiles in about a 40 seconds (could be further reduced with
\includeonly, but I usually don't bother). Or if I'm working on a specific
tikzpicture, I only ever need to rebuild that, which is also quite acceptable (see the workflow notes above).
If anyone wants to have a go at it themselves (or be appalled at some LaTeX heresy I probably committed), the document can be found here (full pdf versions available in the snapshots directory). It's a bit of a convoluted setup because I also use a separate
build directory and getting that to work properly with
tikzexternalize is a bit of a story in and of itself. But if anyone wants to have a look at an actual use-case for a non-tiny project with
tikzexternalize, it might be of interest.