Minted caching in tikz externalize job

Enabled caching options for minted and tikz lead to unexpected and inefficient caching. For every tikz figure that is externalized, all minted code blocks of the main document are cached within the tikz cache directory. For documents with e.g. >50 figures and >100 code listings, caching takes a long time and uses more disk space than would be necessary.

To give an example, consider this MWE:

\documentclass[11pt,a4paper]{article}
\usepackage[T1]{fontenc}
\usepackage[cache]{minted}
\usepackage{pgf,tikz}
\usetikzlibrary{external}
\tikzsetexternalprefix{figures/cache/}
\tikzset{external/system call={pdflatex -shell-escape -halt-on-error -interaction=batchmode -jobname "\image" "\texsource"}}
\tikzexternalize

\begin{document}

\begin{minted}[frame=none,fontsize=\small,linenos=true]{r}
plotdata <- data.frame(this = rnorm(20), that = rnorm(20))
\end{minted}

\begin{tikzpicture}[->,>=stealth',shorten >=1pt,auto,node distance=3cm,
thick,main node/.style={circle,fill=blue!20,draw,font=\sffamily\Large\bfseries}]

\node[main node] (1) {1};
\node[main node] (2) [below left of=1] {2};
\node[main node] (3) [below right of=2] {3};
\node[main node] (4) [below right of=1] {4};

\path[every node/.style={font=\sffamily\small}]
(1) edge node [left] {0.6} (4)
edge [bend right] node[left] {0.3} (2)
edge [loop above] node {0.1} (1)
(2) edge node [right] {0.4} (1)
edge node {0.3} (4)
edge [loop left] node {0.4} (2)
edge [bend right] node[left] {0.1} (3)
(3) edge node [right] {0.8} (2)
edge [bend right] node[right] {0.2} (4)
(4) edge node [left] {0.2} (3)
edge [loop right] node {0.6} (4)
edge [bend right] node[right] {0.2} (1);
\end{tikzpicture}

\end{document}


This leads to the directory structure:

.minted-mwe/
475544BE9F571FFAF1D26ACA3A0AB8D0502F9AF18E3C3312853333EF1520885B.pygtex
default.pygstyle
.minted-figures/
cache/
mwe-figure0/
475544BE9F571FFAF1D26ACA3A0AB8D0502F9AF18E3C3312853333EF1520885B.pygtex
default.pygstyle
figures/
cache/
mwe-figure0.aex
mwe-figure0.dpth
mwe-figure0.log
mwe-figure0.md5
mwe-figure0.pyg


where the *.pygtex files are redundant. I assume this is because of the way tikz does the system call to process the individual figures, i.e. all minted environments are considered in the external pdflatex job.

Any ideas on how to resolve this? Am I missing/misconfiguring something?

This is due to the way that tikz externalization works. If you look in the PGF/TikZ manual, under externalization, under "Details About The Process" (Section 50.4.5, page 625 in the version 3.0.0 manual), you will see that externalize works by compiling the entire file once for each image that is being externalized. During this compiling process, all parts of the document not related to the current image are "silently thrown away". Unfortunately, while minted output is thrown away, minted will still check to see if cached content exists, and if not, create it. Due to the way the compiling works for the externalized images, minted can't detect the existing cache from the main run, and will create a new cache.

There may be things that can be improved from the tikz side; I don't know enough about the technical details of the externalization process. From the minted side, the issue of duplication can be solved by loading minted conditionally (this needs minted 2.0):

\ifcsname tikzexternalrealjob\endcsname
\usepackage[draft]{minted}
\else
\usepackage[cache]{minted}
\fi


If the macro \tikzexternalrealjob exists (if the file is being compiled solely to externalize an image), this will load minted in draft mode. In draft mode, no external files are created and Python (Pygments) will never be called. (Unused cache directories for the images will still be created...I will see about fixing that, and possibly adding tikz externalization detection, in the next minted release: issue tracker.) If \tikzexternalrealjob does not exist, then this is a normal compile, and minted proceeds normally.

• Updated minted to 2.0; runs fine modulo the empty directories. Thanks for the workaround and keep up the good work! – phaebz Apr 24 '15 at 1:21