Script to automate externalizing TikZ graphics

Question

Is there a script to automate externalizing TikZ graphics?

(As suggested by Scott here)

Answer 1

The TikZ 2.10 library, external, addresses this issue. Externalizing all TikZ graphics in a LaTeX document is as easy as:

\usetikzlibrary{external}
\tikzexternalize

If your document is named report.tex, this will dump out a series of images: report-figure0.pdf, report-figure1.pdf, etc, etc. To get a more manageable output, you can specify a directory for in which the files are to be placed and names that are more descriptive than report-figure0.pdf:

\usetikzlibrary{external}
\tikzexternalize[prefix=figures/]

% Rest of preamble
% Begin document, write stuff

% Set a filename for the next tikzpicture.
\tikzsetnextfilename{importantFigure}
\begin{tikzpicture}  % This will be output to figures/importantFigure.pdf
  % Picture code
\end{tikzpicture}

To compile the document, you will need to ensure the figures directory exists and run pdflatex with shell execution enabled:

mkdir figures
pdflatex -shell-escape <tex file>

A whole pile of options is available to customize the way the figures are output. See section 31.4 (this numbering is subject to change!) of the development version documentation for complete info and options. The manual also covers how to obtain EPS output- but this option looks less polished than PDF output.

Answer 2

When I was writing my MSc thesis, I had created all of my figures and diagrams with tikz and pgfplots (using matplotlib2tikz). Keeping all of them within the source was not acceptable, since the compilation was getting really slow, so I was searching for something that would allow me to automatically compile all those figures to pdf files that I could include in my text.

For various reasons, I was not entirely satisfied with any of the other solutions so I wrote a Python script that suited my purpose.

The things you must keep in mind are:

1. You put the script on the same level as your main.tex file
2. You write your tikz code. You place each figure within a file with extension *.tikz. Each file contains only \begin{tikzpicture} ... \end{tikzpicture}.
3. You write the preamble that you want to use with the *.tikz files. This doesn't necessarily contains the same packages as the preamble of your main.tex file. The reason you keep a common preamble for all the figures is to make it easy to make changes in things like fonts etc.
4. You run the script!

The script searches recursively within a folder structure for *.tikz files and compiles them using the specified preamble. Each time the scripts runs, it stores the modification time of each *.tikz file and on subsequent runs it compiles only the newly created files of the files that have been modified since the last run. If you make changes in your preamble, you just pass a command-line argument and it compiles all the *.tikz files.

On linux it works great. I haven't tested it on windows, but I don't see why it shouldn't run. Anyway, even if there is any problem, the fix is going to be really trivial

Answer 3

Here's my alternative; Konrad's answer is probably just as good, and certainly better documented.

for job in `grep -o beginpgfgraphicnamed\{[0-9A-Za-z-]*\} < $1 | sed -e 's/beginpgfgraphicnamed{//' | sed -e 's/}//'`; do
    pdflatex --jobname $job $1;
    rm $job.log;
    rm $job.out;
    rm $job.aux;
done

Answer 4

I’ve just come up with the following. It seems to work quite well (on Unix/Linux and Mac):

SOURCE=$1
for name in $( \
    grep '^[:space:]*\\beginpgfgraphicnamed' $SOURCE | \
    sed s'/[:space:]*\\beginpgfgraphicnamed{\(.*\)}/\1/' \
); do
    pdflatex --jobname=$name $SOURCE
done

An explanation:

It uses grep to filter out all lines that contain \beginpgfgraphicsnamed, but only if there’s only whitespace before the command in the line. This prevents that commented-out commands are also found.

Next, it invokes sed to filter from those lines only the relevant part, i.e. the command argument that’s to be used as the pdflatex jobname. It then uses for to iterate over all these jobnames and calls pdflatex in turn for each of them.

Answer 5

I've made a shell script, which builds PDF files from all my tikz/pgfplots/...-figures. Compared to existing solutions, my approach has some advantages: (I don't know the existing solutions very good., but I hope some of the advantages are real advantages...)

• The build of the individual images only happens if necessary, i.e. if the source or included data or included headers are newer than the (existing?) PDF. If you plot e.g. some data with pgfplots, you can just put a new csv file in the right folder, and in the next compilation of the main document, this is recognized and just the image using this file is rebuilt.

• Another advantage of my solution is that the PDFs are compiled in parallel. But you get noticed if one or multiple pdfs fail to compile including detailed information from the log files, and then the remaining compilation of the main document stops. The number of compilations running in the background can be limited to a number of processes like 4 or 8 (depending on the number of CPU cores you want to use). This is especially helpfull, if one process needs much RAM (in my case, some extensive pgfplots images need > 1GB of RAM during compilation...)

• draft option is recognize to avoid unneccessary compilations. Also, there is a buildall and buildnone option, which might be helpfull in case of changing the headers again and again.

• Not limited tikz, but any packge should be possible, as each image is just a normal, complete .tex document.

• Concerning tikz, I had some problems with text being cut away at the borders of the pictures using standalone. This behavior does not happen with my solution, as I crop the A4-pages with the images with pdfcrop after compilation.

I have got an .sty-file for the script-call like shown in the following. There is a definition for the images-Folder, two example images and the script call and return value handling. \usepackage{imgs} is the first line after \documentclass in my main document, so it's made sure all pdfs are always up to date.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% Input Parameters                                                %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\ProvidesPackage{imgs}
\DefineFamily{imgs}
\DefineFamilyMember{imgs}

\FamilyNumericalKey{imgs}
{buildall}{@Buildall}{%
    {false}{0},
    {true}{1},%
}
\FamilyNumericalKey{imgs}
{buildnone}{@Buildnone}{%
    {false}{0},
    {true}{1},%
}
\RequirePackage{ifthen} 
\provideboolean{bool-draft}     
\setboolean{bool-draft}{false}
\DeclareOption{draft}{\setboolean{bool-draft}{true}}
\DeclareOption*{%
    \PassOptionsToPackage{\CurrentOption}{imgs}%
}
\FamilyProcessOptions{imgs}
\ProcessOptions\relax   

% rest

\RequirePackage{ifplatform}

\newcommand*\imgs{./imgs/}
\newcommand*\imga{imga}
\newcommand*\imgb{imgb}
\newcommand*\blank{ }


\ifcase\@Buildnone
\ifwindows
\else
  \ifthenelse{\boolean{bool-draft}}   
  {}{
    \newcommand*\blank{ }

    \ifcase\@Buildall
      %\immediate\write18{}
    \or
      \immediate\write18{find \imgs\blank -name '*.tex' | sed 's/^\@backslashchar(.*\@backslashchar)\@backslashchar.tex$/\@backslashchar1.pdf/' | xargs rm -f}
    \fi


    %\immediate\write18{bash -x \imgs img.sh \imgs\blank % for debugging
    \immediate\write18{bash    \imgs img.sh \imgs\blank 
    \imga\blank
    \imgb\blank % insert additional images like this...
      ; echo $? > pdfscreated.res
    }

    \newread\myscriptresult
    \immediate\openin\myscriptresult=pdfscreated.res
    \read\myscriptresult to \ScriptResult
    \immediate\closein\myscriptresult


    \ifnum\ScriptResult>0
      \@latexerr{Image Compilation failed}\@eha
      %\stop
    \fi
  }
\fi
\or
\fi

The image is included like this in a (very short) main document:

\documentclass{article}
\usepackage{imgs} % see code above!
\usepackage{header} % may be the same as for images, or different, as you like
\begin{document}
\includegraphics{\imgs\imga.pdf}
\end{document}

An example for an image would be

\documentclass[10pt]{article}
\usepackage{../header} % own header for images, may be the same as for main document or not...
\begin{document}
\thispagestyle{empty}
\begin{tikzpicture}
    %draw sthg...
    \draw (0,0) -- (1,1);
\end{tikzpicture}
\end{document}

The called shell script is shown in the following. It searches in each .tex file for usepackage, input, pgfplotstableread and recognizes also if used files are newer than than an possibly existing PDF.

#!/bin/bash

# first argument: folder to work in
cd $1
shift

# counter for pdfs to create
cnt=0

# relevant programms
texbin="lualatex --shell-escape " # Might be necessary with BIG memory needs
#texbin="pdflatex --shell-escape "
pdfcrop='pdfcrop --hires '

# wait for max number of jobs function
waitforjobs() {
    while test $(jobs -p | wc -w) -ge "$1"; do wait -n; done
}

# loop through all arguments and (re)build PDFs if necessary
while [[ $# > 0 ]] ; do

    bld=0
    cnt=$(($cnt+1))

    # analyze file name
    # $1 = ... regex cut .tex away if there
    tmp=`echo "$1" | sed s/\.tex//`
    tex="$tmp.tex"
    pdf="$tmp.pdf"
    names[$cnt]="$tmp"

    # test if tex file exists
    if [ ! -e $tex ] ; then
        echo "$tex doesn't exists. Maybe forgot to create it?"
        exit 2
    fi

    # search in file for included, read or inputed files....
    included=`grep 'pgfplotstableread' $tex | sed  's/^.*pgfplotstableread{\(.*\....\)}.*$/\1/'`
    libs=`grep 'usepackage' $tex | sed  's/^.*usepackage\(\[.*\]\)\?{\(.*\)}.*$/\2.sty/'`
    graphics=`grep 'includegraphics' $tex | sed  's/^.*includegraphics\(\[.*\]\)\?{\(.*\)}.*$/\2/'`
    inputs=`grep 'input' $tex | sed  's/^.*input{\(.*\)}.*$/\1/'`
    ## add import and include?!?
    # if inputs don't end on \.\w+, append \.tex...
    list=`echo -e "$included\n$libs\n$graphics\n$inputs"`

    # set buld flag dependent on many conditions....
    if [ ! -e $pdf ] ; then
        bld=1
    fi
    for item in ${list//$'\n'/ } ; do
        #if [ -e $item ] ; then # nonexistent items make error in latex, I assume this is preferred...
        if [ $item -nt $pdf ] ; then
            bld=1 
        fi
        #fi
    done
    if [ $tex -nt $pdf ] ; then
        bld=1
    fi

    # no (re)build necessary
    if [ $bld -eq 0 ] ; then
        echo "$pdf is already up to date."
        shift
        continue;
    fi

    # rebuild pdf if necessary 
    waitforjobs 4
    $texbin $tex >/dev/null 2>&1 &
    pids[$cnt]=$!

    shift
done



# now we have all PIDs of running compilation.
# check each and wait if necessary...
errorflag=0
errorstring=
max=$(($cnt+1))


while [[ $cnt>0 ]]; do
    if [ ${pids[$cnt]} ]; then
        wait ${pids[$cnt]}
        if [ 0 -ne $? ] ; then
            # error occured!
            logs="${names[$cnt]}.log $logs"
            errorflag=1
            errorstring="${names[$cnt]} $errorstring"
            pids[$cnt]=
        else
            rm -f ${names[$cnt]}.log
            pdf=${names[$cnt]}.pdf
            $pdfcrop $pdf $pdf &
            pids[$cnt]=$!
        fi;
        rm -f ${names[$cnt]}.aux ${names[$cnt]}.nlo ${names[$cnt]}.nls ${names[$cnt]}-blx.bib ${names[$cnt]}.ilg ${names[$cnt]}.idx ${names[$cnt]}.run.xml ${names[$cnt]}.bcf ${names[$cnt]}.out #${names[$cnt]}.toc
    fi;
    cnt=$(($cnt-1));
done
rm -f Ort.idx

cnt=$(($cnt+1))
while [[ $cnt < $max ]] ; do
    wait ${pids[$cnt]}
    cnt=$(($cnt+1))
done


if [ 0 -ne $errorflag ]; then
    cat $logs
    echo --------------------------------
    echo errors occured in: $errorstring
    exit 1
else
    echo --------------------
    echo - all pdfs created -
    echo --------------------
    exit 0
fi