# Lualatex, Pdflatex fail with \externalize

Here is my problem: I have a 100+ page document with ~100 tikz plots. Some plots contain 9000+ points, and compiling them with pdflatex -shell-escape (or whatever main_memory_size) fails. I can compile it with lualatex though, but it takes forever. I decided to use:

    \usetikzlibrary{external}
\tikzset{external/system call={lualatex \tikzexternalcheckshellescape -shell-                escape -halt-on-error -interaction=nonstopmode -jobname "\image" "\texsource"}}
\tikzexternalize[prefix=TikzFigures/]
\tikzset{external/force remake}


And it fails with

    "16684 words still in use, bla-bla ...may be you did not enable shell-escape         ...bla-bla write18...please refer to picture-17.log",


but there is no error message in that log. If I switch back to pdflatex, with proper adjustments, error log says again that I'm out of memory. I thought lualatex manages memory dynamically, and externalize should work when regular compilation fails, but I got the opposite case. I have 4-core i7 with 8Gb RAM, hardly half is used.. What to do? I use TexNic center and MikTex, so in the editor options of course I hap put the -shell-escape as well as -enable-write18 in all possible combinations for both pdflatex output profile and lualatex profile. I have a main .tex file with several included chapters, with separate tikzpistures in .tex files as well. Moreover, if I delete all the images from the TikzFigures/ folder, now both lualatex and pdflatex fail already at the first image. If I comment out lines responsible for externalization, it compiles. It takes 5 minutes and 500Mb of memory, but it compiles with one warning about scrbook\float@addtolist that it is about to get outdated.

    \documentclass[fontsize=12,BCOR=0mm]{scrbook}
\usepackage{etex}
\usepackage[USenglish]{babel}
\usepackage{url}
\usepackage{natbib}
\usepackage{breakcites}
\usepackage{enumerate}
\usepackage{caption}
\usepackage{nomencl}
\usepackage{setspace}
\usepackage{listings}
\usepackage{booktabs}
\usepackage{tikz}
\usepackage{tikzscale}
\usepackage{pgf}
\usepackage{pgfplots}
\usepackage{subfig}
\pgfplotsset{compat=1.8}
\pgfplotsset{colormap={my_hot}{[1cm]color(0cm)=(black);
color(3cm)=(red);color(4cm)=(orange);color(5cm)=(yellow);color(8cm)=(white)}}

\pgfplotsset{every axis/.append style={line width=1pt,
tick style={tick align=outside, line width=0.6pt}}}
\pgfplotsset{ticks=both}%/.append style={thin}}
\pgfplotsset{ylabel shift=-4pt}
\pgfplotsset{yticklabel style={xshift=0.1cm}}
\pgfplotsset{label style={font=\small}}
\pgfplotsset{every axis legend/.style={inner ysep=1.5pt}}
\usetikzlibrary{automata, arrows}
\usepackage{color}
\definecolor{light-gray}{gray}{0.95}
\lstset{backgroundcolor=\color{light-gray}}%, xleftmargin=\noindent}

\usepackage{multibib}

\usetikzlibrary{external}
\tikzset{external/system call={lualatex \tikzexternalcheckshellescape
-enable-write18 -halt-on-error -interaction=nonstopmode
-jobname "\image" "\texsource"}}
\tikzexternalize[prefix=TikzFigures/]
\tikzset{external/force remake}

\makenomenclature
\noindent\keywordname\enspace\ignorespaces#1}
\setcapindent{0pt}
\renewcommand{\nomname}{List of Symbols}

\makeatletter
\renewenvironment{thebibliography}[1]
{\subsection*{\refname}%
\@mkboth{\MakeUppercase\refname}{\MakeUppercase\refname}
\list{\@biblabel{\@arabic\c@enumiv}}
{\settowidth\labelwidth{\@biblabel{#1}}
\leftmargin\labelwidth
\@openbib@code
\usecounter{enumiv}
\let\p@enumiv\@empty
\renewcommand\theenumiv{\@arabic\c@enumiv}}
\sloppy
\clubpenalty4000
\@clubpenalty \clubpenalty
\widowpenalty4000%
\sfcode\.\@m}
{\def\@noitemerr
{\@latex@warning{Empty thebibliography' environment}}
\endlist}
\makeatother

\makeindex
\begin{document}
\pagenumbering{roman}
\input{title.tex}
\maketitle

\include{abstract}

\include{acknowledgements}

\tableofcontents
\listoftables
\listoffigures

\clearpage
\pagenumbering{arabic}

\include{Abstract}
\include{Introduction}
\include{Algorithms}
\include{Modeling}
\include{Performance}
\include{Conclusions}

\pagestyle{plain}
\cleardoublepage

\setlength{\nomitemsep}{-0.7\parsep}

\begin{spacing}{0.4}
\printnomenclature [14mm]
\end{spacing}

\bibliography{paper-bibfile-short}

\end{document}

! Package tikz Error: Sorry, the system call 'lualatex --shell-escape -intera
ion=nonstopmode -jobname "TikzFigures/phd_thesis-figure17" "\def\tikzexternal
aljob{phd_thesis}\input{phd_thesis}"' did NOT result in a usable output file
ikzFigures/phd_thesis-figure17' (expected one of .pdf:.jpg:.jpeg:.png:). Plea
verify that you have enabled system calls. For pdflatex, this is 'pdflatex -
ell-escape'. Sometimes it is also named 'write 18' or something like that. Or
aybe the command simply failed? Error messages can be found in 'TikzFigures/p
_thesis-figure17.log'. If you continue now, I'll try to typeset the picture.

-
Did you run lualatex --shell-escape yourFile? Can you extend your example to make it compilable, i.e. \documentclass{<theClassYouAreUsing>} all over to \end{document}? – Johannes_B Dec 16 '13 at 18:54
Welcome to TeX.SX! You can have a look at our starter guide to familiarize yourself further with our format. – Martin Schröder Dec 17 '13 at 10:42
Seems that "externalize" simply does not work with lualatex. If I compile the main file with pdftex, with {external/system call={lualatex -... it still fails with "pdftex out of memory" althoug it is supposed to call lualatex as I understand. – Alexander Dec 17 '13 at 13:02

Why not using pdflatex with increasing memory manually.

here is an example (numbers are arbitrary large, you may want try some other values)

  \tikzset{external/system call= {pdflatex -save-size=80000
-pool-size=10000000
-extra-mem-top=50000000
-extra-mem-bot=10000000
-main-memory=90000000
\tikzexternalcheckshellescape
-halt-on-error
-interaction=batchmode
-jobname "\image" "\texsource"}}

-
I just did. Not a single picture produced. And I already solved the issue of 9000 dots in a plot by plotting it with matplotlib and including as a ".png". So this should not be a memory failure now. – Alexander May 17 '14 at 10:16