# lualatex with externalized tikz images runs very slow and often fails

I have a document with more than 60 pages which contains more than 12 pgfplots. I started hitting TeX memory limits fairly early; some of the plots are surface plots based on imported coordinate data. To get around that, I used Christian Feuersänger's externalize library for TiKZ, building all the graphics with lualatex. This worked for a while, but as I add more images, I'm having problems again.

Important detail: I'm using Texmaker and I build with latexmk (running lualatex) from there. I'm also using fontspec. Everything is externalized, I include the following in my preamble:

\usepgfplotslibrary{external}
\tikzset{external/system call={lualatex --shell-escape -halt-on-error -interaction=batchmode -jobname "\image" "\texsource"}}
\tikzset{external/optimize=false}
\tikzexternalize


(I had to turn optimization off because figures were being optimized away; in one every single figure was optimized away. I do not understand why this is happening.)

When I build the document in Texmaker it completes without errors, but I end up with an empty document, and when I look at the main document log, I see errors like this:

===== Image 'Interpolation-figure17' is up-to-date. ======

!LuaTeX error (file Interpolation-figure17.pdf): xpdf: reading PDF image f
ailed
==> Fatal error occurred, no output PDF file produced!


The PDF that causes the error cannot be opened. When I examine the log file, there is this message:

(./Praktische_Arbeiten/MATLAB/LaTeX/delaunay_xy.tex (./coolwarm.map)) (./Prakti
sche_Arbeiten/MATLAB/LaTeX/delaunay.tex (./coolwarm.map)
! Missing $inserted. <inserted text>$
l.7389 \end{axis}

Here is how much of LuaTeX's memory you used:
72271 strings out of 494718
745118,8504848 words of node,token memory allocated
582528 words of node memory still in use:
452 hlist, 24 vlist, 26 rule, 1 ins, 307 disc, 114 math, 18612 glue, 54 kern,
72 penalty, 17 margin_kern, 2718 glyph, 1 page_insert, 77991 attribute, 277 glu
e_spec, 75773 attribute_list, 1 action, 11 temp, 6 if_stack, 1 open, 137 write,
1 close, 17 local_par, 11 dir, 57660 pdf_literal, 1 pdf_start_link, 1 pdf_end_li
nk, 7 pdf_dest, 102 pdf_colorstack, 2 pdf_setmatrix, 2 pdf_save, 2 pdf_restore n
odes
avail lists: 1:127,2:2,3:2159,4:9575,5:51,6:1729,7:95,8:4,9:298,10:681,11:513
,12:229
72945 multiletter control sequences out of 65536+600000
201 fonts using 26599311 bytes
98i,21n,111p,10661b,12912s stack positions out of 5000i,500n,10000p,1000000b,100000s
!  ==> Fatal error occurred, no output PDF file produced!


The last two tikz plots (delaunay.tex, delaunay_xy.tex) result in broken PDFs. If I build those on their own, also using externalize, however, there they compile successfully, so the tikz code itself seems to be ok.

If I start the build from Texmaker and then do an strace on lualatex while it is building those particular figures, I see these lines:

read(43, "6845 5538390\\\\\n};\n\\addplot [\ncol"..., 4096) = 4096
read(43, "rget plot\n]\ntable[row sep=crcr]{"..., 4096) = 4096
read(43, "\nforget plot\n]\ntable[row sep=crc"..., 4096) = 4096


The thing is, those lines are being printed at roughly 0.25 lines per second, which is excruciatingly slow. These are just the read operations! For context, here is an example of an individual plot like the one you see being read in above:

\addplot [
color=blue,
solid,
forget plot
]
table[row sep=crcr]{
559795 5521920\\
559795 5522835\\
558500 5522835\\
559795 5521920\\
};


I can understand that lualatex is slower and that large compiles take time, but I tried a compile of the entire document from scratch this morning and it was not finished after more than an hour; when it does complete I get the errors you see above. The figures compile properly on their own. Memory consumption during a latexmk run is entirely reasonable; I don't see my system swapping, and although one processor core is at 100% I think that's to be expected while compiling anything. Something is clearly wrong here.

I know that one answer is going to be, "just precompile them", but I have a lot of dynamic data here; I'm repeatedly generating new output as I adjust the parameters of my plots. I can't afford to be manually precompiling things and run the risk that the figures don't correspond to each other. I'm not married to the way I am doing it, and as long as it's a one-command solution (ideally something that will run from inside Texmaker and have it show the log) I am open to alternatives.

-
Without optimize, I would expect quadratic runtime for the externalization (i.e. it would be incredibly slow). I would be interested in precise bug reports (preferrably with a minimal working example) to understand why you need to disable the optimization. Note that external mainly affects the time to built stuff, not the peak memory usage. –  Christian Feuersänger Sep 28 at 19:49
Your error message with the "missing $" does not indicate memory issues. Sounds more like a compile problem!? – Christian Feuersänger Sep 28 at 19:51 @ChristianFeuersänger The missing$ turned out to be the result of an axis label with math in it. The optimization failures were the result of some of the pictures having document environments (that explains why they compiled on their own but not in the main document!). Sorry, it's late nights and deadline pressure. In the meantime, perhaps you might like to look at my colormap problem here? tex.stackexchange.com/questions/135547/… I think this one does have to do with a bug in 'pgfplots'. –  Stephen Bosch Sep 29 at 8:20
When using external you can compile your document in parallel processes which reduces the compile time. Here is the ChristianFeuersänger solution to benefit from multi-threaded processing: tex.stackexchange.com/q/136658/24167 –  M.Reza Nov 22 at 12:25