TeX capacity exceeded using pgfplots with/without externalization

Question

The following MWE compiles (for me)

\documentclass{article}

\usepackage{pgfplots}
\pgfplotsset{compat=newest}
\usetikzlibrary{external}
\tikzexternalize[prefix=figures/externalized/]


\begin{document}
\begin{figure}
    \centering
    \begin{tikzpicture}

        \begin{axis}

%           \addplot3[surf,mesh/rows=129,mesh/cols=129,mesh/ordering=y varies] table[x=X,y=Y,z=psi] {figures/data/LidDrivenCavity/Lid_SPIKE_128x128_Re1000_psi.dat};

            \addplot3[contour gnuplot,mesh/rows=129,mesh/cols=129,mesh/ordering=y varies] table[x=X,y=Y,z=psi] {figures/data/LidDrivenCavity/Lid_SPIKE_128x128_Re1000_psi.dat};

        \end{axis}

    \end{tikzpicture}
    \caption{ciao}
\end{figure}
\end{document}

where the data file is here. If I de-comment the first \addplot3 and comment the second one, then I get an error. The final part of the log file (the interesting one, I think) is

===== 'mode=convert with system call': Invoking 'pdflatex -shell-escape -halt-o
n-error -interaction=batchmode -jobname "figures/externalized/prova-figure0" "\
def\tikzexternalrealjob{prova}\input{prova}"' ========
\openout3 = `prova.auxlock'.

runsystem(pdflatex -shell-escape -halt-on-error -interaction=batchmode -jobname
 "figures/externalized/prova-figure0" "\def\tikzexternalrealjob{prova}\input{pr
ova}")...executed.

\openout3 = `prova.auxlock'.


./prova.tex:22: Package tikz Error: Sorry, the system call 'pdflatex -shell-esc
ape -halt-on-error -interaction=batchmode -jobname "figures/externalized/prova-
figure0" "\def\tikzexternalrealjob{prova}\input{prova}"' did NOT result in a us
able output file 'figures/externalized/prova-figure0' (expected one of .pdf:.jp
g:.jpeg:.png:). Please verify that you have enabled system calls. For pdflatex,
 this is 'pdflatex -shell-escape'. Sometimes it is also named 'write 18' or som
ething like that. Or maybe the command simply failed? Error messages can be fou
nd in 'figures/externalized/prova-figure0.log'. If you continue now, I'll try t
o typeset the picture.

See the tikz package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              

l.22    \end{tikzpicture}

? 
PGFPlots: reading {figures/data/LidDrivenCavity/Lid_SPIKE_128x128_Re1000_psi.da
t}
Runaway argument?
{\pgfkeyssetvalue {/pgfplots/execute at begin plot visualization}{}\pgfkeyssetv
alue \ETC.
./prova.tex:22: TeX capacity exceeded, sorry [main memory size=5000000].
<argument> ...5e-1],2Y7.753628e-3]}{2Y8.20894\ETC.

l.22    \end{tikzpicture}

If you really absolutely need more capacity,
you can ask a wizard to enlarge me.


Here is how much of TeX's memory you used:
 20761 strings out of 493054
 547383 string characters out of 6134761
 5000001 words of memory out of 5000000
 23958 multiletter control sequences out of 15000+600000
 3640 words of font info for 14 fonts, out of 8000000 for 9000
 1141 hyphenation exceptions out of 8191
 62i,4n,84p,728b,1519s stack positions out of 5000i,500n,10000p,200000b,80000s
./prova.tex:22:  ==> Fatal error occurred, no output PDF file produced!

similar (or the same) error appears if externalization is not enabled/used.

My TeX distribution is TeX Live on Mac OS X 10.9.5 and I use TeXShop as editor.

Once I (partially) solved a similar problem on Windows+MikTeX+TeXniccenter by modifying/adding options to the command executed when compile button is pressed. I say partially since I actually solved the problem for some data file. Then I experienced the same problem with larger datafile and/or different plots (I don't remember anymore) and didn't find a way out, no matter how much I extended the memory.

What should I do now?

Answer 1

I encountered the same problem with a MikTeX distribution on Windows (using TeXniccenter as editor), the related question I posted being here.

This time, on TeXLive, I solved the problem using LuaLaTeX. I never tried before, since I though it was hard to use, or something much different from LaTeX. On the contrary is it nothing more than another program which can be used in place of pdfLaTeX. So I simply run

lualatex --enable-write18 main.tex

from command line, instead of

pdflatex --enable-write18 main.tex

and everything worked (after so much time needed for lualatex to typeset a large 3D surf plot!) I'm a LaTeX user, more than a LaTeX expert, so this solution was not obvious for me.

If you have to typeset so large tikzpictures it's mandatory to make use the externalize library of tikz. In this case, you can run LuaLaTeX just once, to typeset the picture. With this done, you can fall back on pdfLaTeX which will see that the figures are up-to-date, at least until a change is made to the picture, in which case you should run LuaLaTeX once again to avoid memory problems.