Troubles with hyperref and Tikz externalization

Question

Consider the following example :

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[usenames,dvipsnames]{xcolor}
\usepackage{hyperref}
\usepackage{tikz}
\usepackage{xparse}

\usetikzlibrary{external}
\tikzexternalize[prefix=./]


\NewDocumentCommand{\mypic}{O{red}O{1}}{%
\IfFileExists{mypic_#1.pdf}{\includegraphics[scale=#2]{mypic_#1.pdf}}{%
\tikzsetnextfilename{mypic_#1}%
\begin{tikzpicture}
\draw[#1] (1,0)--(0,1);
\end{tikzpicture}%
}}

\begin{document}

\mypic[Cerulean] \mypic[Cyan] \mypic[red] \mypic[YellowGreen]

\end{document}

The command \mypic is supposed to include a picture whose color is included in the file name if such a file exists, or to create the pdf picture if it was not created before. It is not working on my computer, and I the error message I get is

 Package tikz Error: Sorry, the system call 'pdflatex -shell-escape -halt-on-error -interaction=batchmode -jobname "./mypic_Cerulean"  "\def\tikzexternalrealjob{EIFFEL}\input{EIFFEL}"' did NOT result in a usable output file

However, when removing the hyperrefpackage, it works just fine. I've read a few threads about the interactions of hyperref and TikZ externalization but none of the solutions I've tried worked for me. Is there anything I can do ?

Even stranger : keeping hyperref but removing the \IfFileExists part, it sill works (but does not allow me to change the scale of the pictures). Where does this come from ?

Answer 1

The problem is not directly related to hyperref; the root cause is that you are trying to skip the picture while it is being generated. This results in a 0-size pdf, and that is deleted right-away by pdftex.

This is what happens:

• the main program is typeset, and this is where your "if image exists" logic should apply. It does not since the graphics does not exist.
• the externalization is started. To this end, it starts to generate the picture (let us assume it starts with mypic_Cerulean). Then it checks if mypic_Cerulean.pdf exists -- which is true if the pdf's header has already been written.
• Your 'if' strips the picture code as such, and the externalization results in an empty output file.
• pdftex removes the started pdf file since it contains no output.
• The main program tries to include the file but fails to find it.

Thus, the logic is essentially a programming error in your script: it must not apply the "if image exists" while it is currently about to generate it.

The reason why hyperref makes a difference is that hyperref (appears to) generate some header information in the output pdf, i.e. it causes the file to be written before the first shipout starts. Note that hyperref is not the only tool which generates such header information, but in this case, it makes the difference.

You have to use \tikzifexternalizing in order to fix the problem. It works with the following modification:

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[usenames,dvipsnames]{xcolor}
\usepackage{hyperref}
\usepackage{tikz}
\usepackage{xparse}

\usetikzlibrary{external}
\tikzexternalize[prefix=./]


\NewDocumentCommand{\mypic}{O{red}O{1}}{%
    \tikzifexternalizing{%
        \def\DOIT{1}%
    }{%
        \IfFileExists{mypic_#1.pdf}{%
            \includegraphics[scale=#2]{mypic_#1.pdf}%
            \def\DOIT{0}%
        }{%
            \def\DOIT{1}%
        }%
    }%
    %
    \if1\DOIT
        \tikzsetnextfilename{mypic_#1}%
        \begin{tikzpicture}
        \draw[#1] (1,0)--(0,1);
        \end{tikzpicture}%
    \fi
}

\begin{document}

\mypic[Cerulean] \mypic[Cyan] \mypic[red] \mypic[YellowGreen]

\end{document}

The key idea is to apply your special logic only during the generation of the main .tex file, but omit it if the externalization is running.

The \def\DOIT{<value>} stores <value> into the macro \DOIT. The \if1\DOIT is a 1-char-comparison which checks if \DOIT expands to "1". This check works if and only if \DOIT expands to at most one character (which is satisfied here: it can hold 1 or 0).

Answer 2

I cannot explain the error. Do not give me the bounty. (see next section)

Assuming your MWE, \scalebox works. Let me know if you intend to achieve something more complicated. By the way, whatever your goal is, the logic is that external is superior to \IfFileExists.

\documentclass{article}
\usepackage[usenames,dvipsnames]{xcolor}
\usepackage{tikz,xparse,hyperref}
\usetikzlibrary{external}\tikzexternalize
\NewDocumentCommand{\mypic}{O{red}O{1}}{
    \tikzsetnextfilename{mypic_#1}
    \scalebox{#2}{
        \begin{tikzpicture}
        \draw[#1] (1,0)--(0,1);
        \end{tikzpicture}%
    }
}
\begin{document}
    \mypic[Cerulean]    \mypic[Cyan]    \mypic[red]    \mypic[YellowGreen]
    \mypic[Cerulean][1] \mypic[Cyan][2] \mypic[red][3] \mypic[YellowGreen][4]
\end{document}

What causes the error?

Before I can answer the question, I want to talk about the library external.

Question: How does external work?

Wrong answer:

It extracts the code between \begin{tikzpicture} and \end{tikzpicture}. And then runs that piece of code.

You can prove that this is wrong by adding some global setting outside:

    \tikzset{
        every picture/.style={ ... }
    }

Correct answer:

It calls pdftex to run the original file again, but this time:

• the output routine is suppressed, unless it meets the tikzpicture;
• it passes -jobname "name_of_that_picture" as an option, so this run would end up with name_of_that_picture.pdf.

This mechanism is explained in the manual, currently part IX The Basic Layer section 107 Externalizing Graphics.

So ... what exactly happens normally?

Image that you call pdftex:

• pdftex runs your_article.tex;
• pdftex reaches \mypic{Cerulean};
• pdftex tries \IfFileExists{mypic_Cerulean.pdf}, but in vain;
• pdftex decides to generated mypic_Cerulean.pdf;
• pdftex calls another pdftex -jobname "mypic_Cerulean";
  • pdftex₂ runs your_article.tex;
  • pdftex₂ reaches \mypic{Cerulean};
  • pdftex₂ tries \IfFileExists{mypic_Cerulean.pdf}, but in vain;
  • pdftex₂ decides to typeset the tikzpicture;
  • pdftex₂ finishes, leaves mypic_Cerulean.pdf on the disk;
• pdftex is happy to see the result, mypic_Cerulean.pdf included;
• pdftex reaches \mypic{Cyan};
• pdftex tries \IfFileExists{mypic_ Cyan.pdf}, but in vain;
• pdftex decides to generated mypic_ Cyan.pdf;
• pdftex calls another pdftex -jobname "mypic_ Cyan";
  • pdftex₃ runs your_article.tex;

and so on...

So ... why hyperref causes the error?

• pdftex runs your_article.tex;
• pdftex includes hyperref, your_article.pdf created;
• pdftex reaches \mypic{Cerulean};
• pdftex tries \IfFileExists{mypic_Cerulean.pdf}, but in vain;
• pdftex decides to generated mypic_Cerulean.pdf;
• pdftex calls another pdftex -jobname "mypic_Cerulean";
  • pdftex₂ runs your_article.tex;
  • pdftex₂ includes hyperref, mypic_Cerulean.pdf created;
  • pdftex₂ reaches \mypic{Cerulean};
  • pdftex₂ tries \IfFileExists{mypic_Cerulean.pdf}, successes;
  • pdftex₂ decides to include mypic_Cerulean.pdf, but in vain;
  • pdftex₂ terminates with Fatal error;
• pdftex is unhappy to see no mypic_Cerulean.pdf;
• pdftex pauses and throws the error you saw;

Other workarounds

Try not to test the file by yourself. If you really want to do so, mimic the way PGF testing files. If you just want to control \includegraphics, PGF provides /pgf/images/include external.