Lualatex, Pdflatex fail with \externalize

Question

Here is my problem: I have a 100+ page document with ~100 tikz plots. Some plots contain 9000+ points, and compiling them with pdflatex -shell-escape (or whatever main_memory_size) fails. I can compile it with lualatex though, but it takes forever. I decided to use:

    \usetikzlibrary{external}
    \tikzset{external/system call={lualatex \tikzexternalcheckshellescape -shell-                escape -halt-on-error -interaction=nonstopmode -jobname "\image" "\texsource"}}
    \tikzexternalize[prefix=TikzFigures/]
    \tikzset{external/force remake}

And it fails with

    "16684 words still in use, bla-bla ...may be you did not enable shell-escape         ...bla-bla write18...please refer to picture-17.log", 

but there is no error message in that log. If I switch back to pdflatex, with proper adjustments, error log says again that I'm out of memory. I thought lualatex manages memory dynamically, and externalize should work when regular compilation fails, but I got the opposite case. I have 4-core i7 with 8Gb RAM, hardly half is used.. What to do? I use TexNic center and MikTex, so in the editor options of course I hap put the -shell-escape as well as -enable-write18 in all possible combinations for both pdflatex output profile and lualatex profile. I have a main .tex file with several included chapters, with separate tikzpistures in .tex files as well. Moreover, if I delete all the images from the TikzFigures/ folder, now both lualatex and pdflatex fail already at the first image. If I comment out lines responsible for externalization, it compiles. It takes 5 minutes and 500Mb of memory, but it compiles with one warning about scrbook\float@addtolist that it is about to get outdated.

    \documentclass[fontsize=12,BCOR=0mm]{scrbook}
    \usepackage{etex}
    \usepackage[USenglish]{babel}
    \usepackage{url}
    \usepackage{natbib}
    \usepackage{breakcites}
    \usepackage{enumerate}
    \usepackage{caption}
    \usepackage{nomencl}
    \usepackage{setspace}
    \usepackage{listings}
    \usepackage{booktabs}
    \usepackage{tikz}
    \usepackage{tikzscale}
    \usepackage{pgf}
    \usepackage{pgfplots}
    \usepackage{subfig}
    \pgfplotsset{compat=1.8}
    \pgfplotsset{colormap={my_hot}{[1cm]color(0cm)=(black);
    color(3cm)=(red);color(4cm)=(orange);color(5cm)=(yellow);color(8cm)=(white)}}

    \pgfplotsset{every axis/.append style={line width=1pt, 
    tick style={tick align=outside, line width=0.6pt}}}
    \pgfplotsset{ticks=both}%/.append style={thin}}
    \pgfplotsset{ylabel shift=-4pt}
    \pgfplotsset{yticklabel style={xshift=0.1cm}}
    \pgfplotsset{label style={font=\small}}
    \pgfplotsset{every axis legend/.style={inner ysep=1.5pt}}
    \usetikzlibrary{automata, arrows}
    \usepackage{color}
    \definecolor{light-gray}{gray}{0.95}
    \lstset{backgroundcolor=\color{light-gray}}%, xleftmargin=\noindent} 


    \usepackage{multibib}


    \usetikzlibrary{external}
    \tikzset{external/system call={lualatex \tikzexternalcheckshellescape 
    -enable-write18 -halt-on-error -interaction=nonstopmode 
    -jobname "\image" "\texsource"}}
    \tikzexternalize[prefix=TikzFigures/]
    \tikzset{external/force remake}

    \makenomenclature
    \newcommand{\keywords}[1]{\par\addvspace\baselineskip
    \noindent\keywordname\enspace\ignorespaces#1}
    \setcapindent{0pt} 
    \renewcommand{\nomname}{List of Symbols}

    \makeatletter 
    \renewenvironment{thebibliography}[1]
     {\subsection*{\refname}%
      \@mkboth{\MakeUppercase\refname}{\MakeUppercase\refname}
      \list{\@biblabel{\@arabic\c@enumiv}}
    {\settowidth\labelwidth{\@biblabel{#1}}
    \leftmargin\labelwidth
    \advance\leftmargin\labelsep
    \@openbib@code
    \usecounter{enumiv}
    \let\p@enumiv\@empty
    \renewcommand\theenumiv{\@arabic\c@enumiv}}
      \sloppy
      \clubpenalty4000
      \@clubpenalty \clubpenalty
      \widowpenalty4000%
      \sfcode`\.\@m}
     {\def\@noitemerr
       {\@latex@warning{Empty `thebibliography' environment}}
      \endlist} 
    \makeatother

    \makeindex
    \begin{document}
    \pagenumbering{roman}
    \input{title.tex}
    \maketitle


    \include{abstract}

    \include{acknowledgements}

    \tableofcontents
    \listoftables 
    \listoffigures 

    \clearpage
    \pagenumbering{arabic}

    \include{Abstract}
    \include{Introduction}
    \include{Algorithms}
    \include{Modeling}
    \include{Performance}
    \include{Conclusions}


    \pagestyle{plain}
    \cleardoublepage
    \addcontentsline{toc}{chapter}{List of Symbols}

    \setlength{\nomitemsep}{-0.7\parsep}

    \begin{spacing}{0.4}
    \printnomenclature [14mm]
    \end{spacing}

    \bibliography{paper-bibfile-short}


    \end{document}

! Package tikz Error: Sorry, the system call 'lualatex --shell-escape -intera
ion=nonstopmode -jobname "TikzFigures/phd_thesis-figure17" "\def\tikzexternal
aljob{phd_thesis}\input{phd_thesis}"' did NOT result in a usable output file
ikzFigures/phd_thesis-figure17' (expected one of .pdf:.jpg:.jpeg:.png:). Plea
verify that you have enabled system calls. For pdflatex, this is 'pdflatex -
ell-escape'. Sometimes it is also named 'write 18' or something like that. Or
aybe the command simply failed? Error messages can be found in 'TikzFigures/p
_thesis-figure17.log'. If you continue now, I'll try to typeset the picture.

Answer 1

Why not using pdflatex with increasing memory manually.

here is an example (numbers are arbitrary large, you may want try some other values)

  \tikzset{external/system call= {pdflatex -save-size=80000 
                           -pool-size=10000000 
                           -extra-mem-top=50000000 
                           -extra-mem-bot=10000000 
                           -main-memory=90000000 
                           \tikzexternalcheckshellescape 
                           -halt-on-error 
                           -interaction=batchmode
                           -jobname "\image" "\texsource"}}