3

I am writing a paper, which includes several large tikz figures and I am having a lot of problems. I read a lot of posts about this, but no approach seems to work. I am working with texstudio 4.5.2 and I am using Miktex.

I tried different things, so I will explain chronological. At the end, I summarize the problems. You also find a MWE at the end, which captures the last try, which ends in an error.

In my paper, I would like to add some large figures, which should also be used in other documents with different format settings. At the beginning, I started using separate .tikz-files by \input{} with the following PDFlatex-configuration in texstudio:

pdflatex.exe -synctex=1 -interaction=nonstopmode %.tex

but I received the following error message:

TeX capacity exceeded, sorry [main memory size=5000000]. ...city=0.7 ] ({x},{y},{testfunction(x, y)});

Then I changed the PDFlatex configuration to:

pdflatex.exe -synctex=1 -interaction=nonstopmode --extra-mem-top=10000000 %.tex

This worked, so my main document is compiled. But the new problem is, that each compilation oif the main file takes very long time, since each large tikz figure has to be compiled once again.

Therefore, I tried to use standalone tikz figures and import the .pdfs. But the problem is, that size and font size don't match my document. I know, that I could change the standalone settings, such that these properties match with the main document. But I would always have to change the settings, if I want to use the figures in other documents with different styles (one column, two columns...), what is annoying and not comfortable.

Therefore, I changed my tikz-settings to enable externalization in the preamble. I also adapted the PDFlatex-configuration once again to:

pdflatex.exe -synctex=1 -interaction=nonstopmode --extra-mem-top=10000000 --shell-escape %.tex

As long as the figures are small (e.g. by setting samples=10), externalization seems to work normal. But when I increase the samples (e.g. samples=100), I get an error message, even if it compiles:

Package tikz Error: Sorry, the system call 'pdflatex -shell-escape -halt-on-e

If I recompile without any changes, it takes the same time as before, which means that externalizing the .tikz-files does not work or has no advantage?

So summurized:

  • increasing memory by adding --extra-mem-top=10000000 worked for large standalone files, but is not usefull, since main document settings might change and figures do not appear as if compiled with \input{.tikz}
  • externalizing seems to work, as long as .tikz figures are small. When they get too large, the error Package tikz Error: Sorry, the system call 'pdflatex -shell-escape -halt-on-error occurs.
  • externalizing does not seem to speed up compilation for large figures.

So what am I doing wrong? How would you work with several large figures?

Below you find the main file:

\documentclass[conference]{IEEEtran}
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize[prefix=figures/] %Pfad, wo die Daten abgelegt werden sollen
\usepackage{pgfplots}
\pgfplotsset{compat=1.18}
\usetikzlibrary{pgfplots.groupplots}
\usepgfplotslibrary{colormaps}%

\usepackage{lipsum}  


\begin{document}    
    \section{Introduction} 
    \lipsum[2]
    
    \begin{figure}[h]
        \input{./figures/large_tikz_figure.tikz}
        \caption{Tikz with input }
    \end{figure}
    
    \begin{figure}[h]
        \includegraphics[width=\columnwidth]{./figures/large_tikz_figure_standalone.pdf}
        \caption{standalone tikz pdf}
    \end{figure}

\end{document}

The large_tiks_figure_standalone.tex

\documentclass[crop,tikz]{standalone}% 
\usepackage{pgfplots}
\usepgfplotslibrary{colormaps}%

\pgfmathdeclarefunction{testfunction}{2}{%
    \pgfmathparse{#1*#1+#2*#2}%
}
\pgfplotsset{
    colormap={whiteblue}{color(0)=(white); color(0.1)=(blue)}
}

\begin{document}
\begin{tikzpicture}
    \begin{axis}[
        view={45}{45},
        xlabel={$x$},
        ylabel={$y$},
        zlabel={$z$},
        xmin=-3,
        ymin=-3,
        xmax=3,
        ymax=3,
        zmin=0,
        zmax=5,
        axis on top=true,
        clip mode=individual,
        ticks=none,
        width=\columnwidth]
        
        %plot test function
        \addplot3 [surf, colormap name=whiteblue, samples=100,domain=-3:3,shader=interp, fill opacity=0.7 ] ({x},{y},{testfunction(x, y)});
        
        \node[ left ,align=center] at (1,1) {test};
    \end{axis}
\end{tikzpicture}
\end{document}

and the extern large_tikz_figure.tikz

\pgfmathdeclarefunction{testfunction}{2}{%
    \pgfmathparse{#1*#1+#2*#2}%
}
\pgfplotsset{
    colormap={whiteblue}{color(0)=(white); color(0.1)=(blue)}
}

\begin{tikzpicture}
    \begin{axis}[
        view={45}{45},
        xlabel={$x$},
        ylabel={$y$},
        zlabel={$z$},
        xmin=-3,
        ymin=-3,
        xmax=3,
        ymax=3,
        zmin=0,
        zmax=5,
        axis on top=true,
        clip mode=individual,
        ticks=none,
        width=\columnwidth] % \columnwidth does not seem to work here?
        
        %plot test function
        \addplot3 [surf, colormap name=whiteblue, samples=100,domain=-3:3,shader=interp, fill opacity=0.7 ] ({x},{y},{testfunction(x, y)});
        
        \node[ left ,align=center] at (1,1) {test};
    \end{axis}
\end{tikzpicture}

And the error from the log file


===== 'mode=convert with system call': Invoking 'pdflatex -shell-escape -halt-o
n-error -interaction=batchmode -jobname "figures/main-figure0" "\def\tikzextern
alrealjob{main}\input{main}"' ========
runsystem(pdflatex -shell-escape -halt-on-error -interaction=batchmode -jobname
 "figures/main-figure0" "\def\tikzexternalrealjob{main}\input{main}")...execute
d.

\openout3 = `main.auxlock'.


! Package tikz Error: Sorry, the system call 'pdflatex -shell-escape -halt-on-e
rror -interaction=batchmode -jobname "figures/main-figure0" "\def\tikzexternalr
ealjob{main}\input{main}"' did NOT result in a usable output file 'figures/main
-figure0' (expected one of .pdf:.jpg:.jpeg:.png:). Please verify that you have 
enabled system calls. For pdflatex, this is 'pdflatex -shell-escape'. Sometimes
 it is also named 'write 18' or something like that. Or maybe the command simpl
y failed? Error messages can be found in 'figures/main-figure0.log'. If you con
tinue now, I'll try to typeset the picture.

See the tikz package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.34 \end{tikzpicture}
                      
This error message was generated by an \errmessage
command, so I can't give any explicit help.
Pretend that you're Hercule Poirot: Examine all clues,
and deduce the truth by order and method.

LaTeX Font Info:    External font `cmex10' loaded for size
(Font)              <7> on input line 1.
LaTeX Font Info:    External font `cmex10' loaded for size
(Font)              <5> on input line 1.
)
\openout3 = `figures/main-figure1.md5'.

\openout3 = `main.auxlock'.


===== 'mode=convert with system call': Invoking 'pdflatex -shell-escape -halt-o
n-error -interaction=batchmode -jobname "figures/main-figure1" "\def\tikzextern
alrealjob{main}\input{main}"' ========
runsystem(pdflatex -shell-escape -halt-on-error -interaction=batchmode -jobname
 "figures/main-figure1" "\def\tikzexternalrealjob{main}\input{main}")...execute
d.

\openout3 = `main.auxlock'.


! Package tikz Error: Sorry, the system call 'pdflatex -shell-escape -halt-on-e
rror -interaction=batchmode -jobname "figures/main-figure1" "\def\tikzexternalr
ealjob{main}\input{main}"' did NOT result in a usable output file 'figures/main
-figure1' (expected one of .pdf:.jpg:.jpeg:.png:). Please verify that you have 
enabled system calls. For pdflatex, this is 'pdflatex -shell-escape'. Sometimes
 it is also named 'write 18' or something like that. Or maybe the command simpl
y failed? Error messages can be found in 'figures/main-figure1.log'. If you con
tinue now, I'll try to typeset the picture.

See the tikz package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.68        \end{tikzpicture}
                        
(That was another \errmessage.)


LaTeX Warning: `h' float specifier changed to `ht'.

<./figures/large_tikz_figure_standalone.pdf, id=12, 322.09235pt x 252.676pt>
File: ./figures/large_tikz_figure_standalone.pdf Graphic file (type pdf)
<use ./figures/large_tikz_figure_standalone.pdf>
Package pdftex.def Info: ./figures/large_tikz_figure_standalone.pdf  used on in
put line 73.
(pdftex.def)             Requested size: 252.0pt x 197.69565pt.

LaTeX Warning: `h' float specifier changed to `ht'.


** Conference Paper **
Before submitting the final camera ready copy, remember to:

 1. Manually equalize the lengths of two columns on the last page
 of your paper;

 2. Ensure that any PostScript and/or PDF output post-processing
 uses only Type 1 fonts and that every step in the generation
 process uses the appropriate paper size.

[1

{C:/Users/\Userx/AppData/Local/MiKTeX/fonts/map/pdftex/pdftex.map}{C:/Users/
\Userx/AppData/Local/Programs/MiKTeX/fonts/enc/dvips/base/8r.enc} <./figures
/large_tikz_figure_standalone.pdf>] (main.aux)
 ***********
LaTeX2e <2023-06-01> patch level 1
L3 programming layer <2023-06-30>
 ***********
 ) 
Here is how much of TeX's memory you used:
 24417 strings out of 475550
 739689 string characters out of 5760645
 6321070 words of memory out of 9390832
 45262 multiletter control sequences out of 15000+600000
 584144 words of font info for 81 fonts, out of 8000000 for 9000
 1141 hyphenation exceptions out of 8191
 102i,12n,107p,727b,2742s stack positions out of 10000i,1000n,20000p,200000b,200000s
4
  • 3
    there have been some new packages for externalization sent to ctan recently (e.g. memoize), (I haven't tried them, so I have no idea how good they are). But generally I wouldn't put to much weight on "confortability". Creating and handling complex images need a certain amount of organization. I would for example put the tikzpicture in some external file, and write a suitable standalone wrapper for every project. Oct 13, 2023 at 9:13
  • Welcome! You can change the command used when compiling the external images. So you might need to increase memory there, for example. But look in the log produced by the external compilation to see if that's the problem. The main log file (which you showed part of above) tells you which log file contains details of why externalisation failed for a particular picture.
    – cfr
    Oct 13, 2023 at 16:34
  • Thanks @UlrikeFischer. I'd been hoping memoize would make it to CTAN soon.
    – cfr
    Oct 13, 2023 at 19:22
  • Thank you all, the solution was to change the compiling command for the external images :) Oct 18, 2023 at 7:33

3 Answers 3

6

A few caveats:

  • --extra-mem-top is not an option for the version of pdfTeX I have (TeX Live) and I could not get your example to compile that way, so I used LuaTeX;
  • your definition of testfunction seems to work, but it does not obviously correspond to the documented use of \pgfmathdeclarefunction and it seemed inessential to the problem, so I switched it in the .tikz case (but not in the standalone case).

Answer:

You can, in fact, combine standalone with externalisation. If you load the standalone package in your main file, you can \input the standalone .tex file. This will ensure that your global settings are consistent, while allowing you to compile the file independently when necessary. standalone will simply discard the preamble unless you specify otherwise.

However, standalone treats document as a group. So you cannot define something in one such file and then use it later in the document. But doing that is bad practice anyway, in my opinion. I would either put such definitions in the main preamble, load them from a separate file or define them in the standalone files' preambles and tell standalone to figure out what's needed by processing those preambles (subpreambles, standalone calls them).

As Ulrike Fischer mentioned, there are a couple of new externalisation packages now available. For this answer, I used memoize. Please understand that I only began playing with this yesterday. Nonetheless, it has so far externalised 30+ images which defeated TikZ's externalisation library.

Downsides:

  • memoize is very new;
  • your document will look rather odd after first compilation;
  • it takes a little more work to set up correctly than using the external library;
  • it may not yet be available in MikTeX, which is usually a bit slower than TeX Live to provide updates;
  • the two most secure of the three extraction options require installing additional software which is not part of your TeX installation (easy on Linux, but possibly less so on Windows?).

Upsides:

  • its author wrote forest and has a track record of providing excellent support;
  • its perl and python extraction scripts have passed the security test for inclusion in the small list of commands engines can run with restricted shell escape (i.e. without the full shell escape needed otherwise);
  • it is faster than TikZ's externalisation;
  • it can externalise images external can't;
  • it names externalised files by checksums rather than in simple series, so if you insert an image at the start of your document, it will compile that image but leave the rest alone whereas, out-of-the-box, external would recompile everything;
  • it is general and not limited to images;
  • it handles label/ref automatically;
  • it aborts without error if it finds a problem and it knows, for example, that remember picture means a tikzpicture shouldn't be externalised;
  • it does not necessarily require shell escape (I did not use shell escape to externalise your plots);
  • because it compiles externalised objects as part of a single compilation of the document - it does not spawn a separate compilation process for each object - it is not only fast, it doesn't 'optimise away' things the object might need.

That is, it is faster, more secure and more robust once setup, but demands a little bit more work to get going.

I used the default memoize settings, which means the perl extraction script. This works without shell escape. On the first run, your document looks rather weird with all kinds of extra pages in funny sizes. On the next run, the extras disappear and the 'externs' (externalised things) are created, by extracting them from the PDF created on the first run. These are then included in the document as necessary and only recreated if the code changes (or they include a \ref which has changed or you ask etc.).

[Can you tell I've been waiting for this package to hit CTAN?!]

\begin{filecontents}[overwrite]{\jobname-large_tikz_figure_standalone.tex}
\documentclass[crop,tikz]{standalone}% 
\usepackage{pgfplots}
\usepgfplotslibrary{colormaps}%

\pgfmathdeclarefunction{testfunction}{2}{%
    \pgfmathparse{#1*#1+#2*#2}%
}
\pgfplotsset{
    colormap={whiteblue}{color(0)=(white); color(0.1)=(blue)}
}

\begin{document}
\begin{tikzpicture}
    \begin{axis}[
        view={45}{45},
        xlabel={$x$},
        ylabel={$y$},
        zlabel={$z$},
        xmin=-3,
        ymin=-3,
        xmax=3,
        ymax=3,
        zmin=0,
        zmax=5,
        axis on top=true,
        clip mode=individual,
        ticks=none,
        width=\columnwidth]
        %plot test function
        \addplot3 [surf, colormap name=whiteblue, samples=100,domain=-3:3,shader=interp, fill opacity=0.7 ] ({x},{y},{testfunction(x, y)});
        
        \node[ left ,align=center] at (1,1) {test};
    \end{axis}
\end{tikzpicture}
\end{document}
\end{filecontents}
\begin{filecontents}[overwrite]{\jobname-large_tikz_figure.tikz}
\begin{tikzpicture}
    \begin{axis}[
        view={45}{45},
        xlabel={$x$},
        ylabel={$y$},
        zlabel={$z$},
        xmin=-3,
        ymin=-3,
        xmax=3,
        ymax=3,
        zmin=0,
        zmax=5,
        axis on top=true,
        clip mode=individual,
        ticks=none,
        width=\columnwidth] % \columnwidth does not seem to work here?
        %plot test function
        \addplot3 [surf, colormap name=whiteblue, samples=100,domain=-3:3,shader=interp, fill opacity=0.7 ] {x^2+y^2};        
        \node[ left ,align=center] at (1,1) {test};
    \end{axis}
\end{tikzpicture}
\end{filecontents}


\documentclass[conference]{IEEEtran}
\usepackage{memoize}
\usepackage{tikz}
\usepackage{standalone}
\usepackage{pgfplots}
\pgfplotsset{compat=1.18}
\usetikzlibrary{pgfplots.groupplots}
\usepgfplotslibrary{colormaps}%
\pgfplotsset{
  colormap={whiteblue}{color(0)=(white); color(0.1)=(blue)}
}
\pgfmathdeclarefunction{testfunction}{2}{%
  \pgfmathparse{#1*#1+#2*#2}%
}
\usepackage{lipsum}  

\begin{document}    
    \section{Introduction} 
    \lipsum[2]
    
    \begin{figure}[htp]% using h alone is a bad idea and pointless as LaTeX overrides it anyway
        \input{\jobname-large_tikz_figure.tikz}
        \caption{Tikz with input }
    \end{figure}
    
    \begin{figure}[htp]
        \input{\jobname-large_tikz_figure_standalone}
        \caption{standalone tikz}
    \end{figure}
\end{document}

two externalised 'externs' extracted by memoize

2
  • Thank you for your comments. Using another externalization lib seems a good idea. But since my collegues have also to work with my document, I prefer not to change to much, so that they are able to compile without additional settings. But for my thesis, I will check this. Oct 18, 2023 at 7:23
  • +1: Excellent entry point for this new package! Feb 18 at 9:08
3

This configuration works for me. (windows, TeXstudio, MiKTeX)

(1) Using lualatex.exe -synctex=1 -interaction=nonstopmode -shell-escape %.tex in MiKTeX.

(2) TestExternalize.tex in the working directory WorkingDir with a subdirectory figures

b

In the code it was added the package l3benchmark to measure the execution time: measure the compilation time

The other lines added to your code are not strictly necessary but help place the auxiliary files created by the externalization in the figures directory and name them using \tikzsetnextfilename{..} before each \begin{figure}. This is important if your document has many tikz figures.

%%% File TestExternalize.tex

% !TeX TS-program = lualatex

\RequirePackage{l3benchmark}% added https://tex.stackexchange.com/a/505840/161015
\ExplSyntaxOn
\AtEndDocument { \benchmark_toc: }
\benchmark_tic:
\ExplSyntaxOff
    
\documentclass[conference]{IEEEtran}

\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize[prefix=./figures/] %changed + Folder needs to be created before compiling <<<<<<

\tikzset{external/system call={% added <<<<<<<<<<<<<<
        lualatex \tikzexternalcheckshellescape
        -halt-on-error -shell-escape -interaction=batchmode
        -jobname "\image" "\texsource"}}


\usepackage{pgfplots}
\pgfplotsset{compat=1.18}
\usetikzlibrary{pgfplots.groupplots}
\usepgfplotslibrary{colormaps}%

\usepackage{lipsum} 

\begin{document}    
    \section{Introduction} 
    \lipsum[2]
    
    \begin{figure}[ht]
        \centering
        \input{./figures/large_tikz_figure.tikz}
        \caption{Tikz with input }
    \end{figure}

    \lipsum
    
\end{document}

(3) Directory structure after compiling.

d

(4) Output: TestExternalize.pdf

e

First run with 100 samples (from the Messages windows)

This is LuaHBTeX, Version 1.15.1 (MiKTeX 22.10) (format=lualatex 2023.7.25)  13 OCT 2023

(./figures/large_tikz_figure.tikz
===== 'mode=convert with system call': Invoking 'lualatex -shell-escape -halt-on-error -interaction=batchmode -jobname "./figuresTestExternalize-figure0" "\def\tikzexternalrealjob{TestExternalize}\input{TestExternalize}"' ========
lualatex: security risk: running with elevated privileges This is LuaHBTeX, Version 1.15.1 (MiKTeX 22.10) system commands enabled.
)

l3benchmark) + TOC: 14.1 s

Second run

(./figures/large_tikz_figure.tikz
===== Image './figuresTestExternalize-figure0' is up-to-date. ======
)

(l3benchmark) + TOC: 1.29 s

See also a similar answer externalize

In TestExternalize-figure0.log you will find

(l3benchmark) + TOC: 11.7 s
1
  • Thank you for your answer. My slution is based on your hints, but I was able to adapt it to pdflatex. Oct 18, 2023 at 7:24
2

thank you all for your replies. My answer is a little bit delayed, since I tried different solutions and I will summarize my findings.

As mentioned twice, switching to lualatex with the settings lualatex.exe -synctex=1 -interaction=nonstopmode -shell-escape %.tex

and the additional

\tikzset{external/system call={% added <<<<<<<<<<<<<<
    lualatex \tikzexternalcheckshellescape
    -halt-on-error -shell-escape -interaction=batchmode
    -jobname "\image" "\texsource"}}

did fix the problem, it compiles fast and a recompilation does not recompile the figures. In my original (project) document I had to add minor changes in the preamble:

\pdfcompresslevel=1
\pdfobjcompresslevel=1

to

\edef\pdfcompresslevel{\pdfvariable compresslevel}
\pdfcompresslevel=1

Since it is necessary, that my collegues can also run the document and we use the same (extern) preamble document for all projects, lualatex may not be sufficient, even if I would prefer this solution!

What finally fixed my problem, when using pdflatex, was adding the \tikset with --extra-mem-top=10000000 in the preamble. The compilation command is: pdflatex.exe -synctex=1 -interaction=nonstopmode --extra-mem-top=10000000 --shell-escape %.tex So, the main document looks like

\documentclass[conference]{IEEEtran}

\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize[prefix=figures/] %Pfad, wo die Daten abgelegt werden sollen
\usepackage{pgfplots}
\pgfplotsset{compat=1.18}
\usetikzlibrary{pgfplots.groupplots}
\usepgfplotslibrary{colormaps}%

\usepackage{lipsum}  


\tikzset{external/system call={% added <<<<<<<<<<<<<<
            pdflatex \tikzexternalcheckshellescape
            -halt-on-error -shell-escape -interaction=batchmode 
            --extra-mem-top=10000000 -jobname "\image" "\texsource"}}

\begin{document}
    
\begin{abstract}
    \lipsum[1]
\end{abstract}
    
    \begin{IEEEkeywords}
blablabla
    \end{IEEEkeywords}
    
    \section{Introduction} 
    \lipsum[2]
    
    \tikzsetnextfilename{Tikz_with_input}
    \begin{figure}[h]
        \input{./figures/large_tikz_figure.tikz}
        \caption{Tikz with input}
    \end{figure}
    
        
    \pgfmathdeclarefunction{testfunction}{2}{%
        \pgfmathparse{#1*#1+#2*#2}%
    }
    %
    \pgfplotsset{
        colormap={whiteblue}{color(0)=(white); color(0.1)=(blue)}
    }
    
    \tikzsetnextfilename{test2}
    \begin{figure}[h]
        \begin{tikzpicture}
            \begin{axis}[
                view={45}{45},
                xlabel={$x$},
                ylabel={$y$},
                zlabel={$z$},
                xmin=-3,
                ymin=-3,
                xmax=3,
                ymax=3,
                zmin=0,
                zmax=5,
                axis on top=true,
                clip mode=individual,
                ticks=none,
                width=\columnwidth] 
                
                %plot test function
                \addplot3 [surf, colormap name=whiteblue, samples=100,domain=-3:3,shader=interp, fill opacity=0.7 ] ({x},{y},{testfunction(x, y)});
                
                \node[ left ,align=center] at (1,1) {test};
            \end{axis}
        \end{tikzpicture}
        \caption{Tikz in main}
    \end{figure}
    
\end{document}
2
  • You might want to add that this solution will only work for MikTeX. As far as I can tell, there's no such option for TeX Live.
    – cfr
    Oct 18, 2023 at 11:45
  • Thanks for summarizing your findings! Feb 18 at 9:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .