pgfplots external with large dataset (lualatex?)

Question

I have fairly large datasets with densely strucutred data that i cant downsampe much without loosing structure. I use TexMaker on Ubuntu 14.04LTS. I use

\usepackage{tikz,pgfplots,pgfplotstable}
\pgfplotsset{compat=1.9}
\usepgfplotslibrary{external}
\tikzexternalize

and

\tikzsetnextfilename{figs/test1}

to externalize the figures. This works good compiling with pdflatex -shell-escape. But as the data becomes to large I am out of memory. I am trying to compile using LuaLaTeX -shell-escape instead of pdfLatex but get the following error message:

! Package tikz Error: Sorry, the system call 'pdflatex -halt-on-error -interact
ion=batchmode -jobname "figs/test1" "\def\tikzexternalrealjob{article}\input{ar
ticle}"' did NOT result in a usable output file 'figs/test1' (expected one of .
pdf:.jpg:.jpeg:.png:). Please verify that you have enabled system calls. For pd
flatex, this is 'pdflatex -shell-escape'. Sometimes it is also named 'write 18'
or something like that. Or maybe the command simply failed?

so pgfPlots seem to call pdfLatex without the appropriate -shell-escape flag and without me specifying it. How can i get this to compile in LuaLatex? or can i compile only the figures in LuaLatex and the rest in pdfLatex (the opposite of what is happening now)? I found this and this example but neither helped me...

Thankfull for help and i am also thankful for any general advice that can help me resolve the problem of ploting larger data sets in pgfplots.

Cheers /J

Answer 1

So i kind of solved the problem with some help form the pgfplots manual pages 470-2, and this other question. I have this in my preamble:

\usepackage{tikz,pgfplots,pgfplotstable}
\pgfplotsset{compat=1.9}
%\usetikzlibrary{external}
\usepgfplotslibrary{external}
\tikzexternalize

\tikzset{external/system call={lualatex -shell-escape -halt-on-error -interact
ion=batchmode -jobname "\image" "\texsource"}}
\usepackage{luaotfload}
\usepackage[EU2]{fontenc}
\usepackage{lmodern}

and i compile with lualatex -shell-escape article.tex from the command line or adding -shell-escape to the lualatex compile option in texMaker so that it becomes lualatex -shell-escape -interaction=nonstopmode %.tex. Another helpful thing to ajust the bit size of the image to balance compile time, even when compiling only once, is to use gnuplot in pgfplots and using the every statement to reduce the sampling of the data. Like this:

\addplot gnuplot 
            [raw gnuplot, mark=none, color=red] 
            {plot '../0/cf/cf3p-bw/cross.dat' every 10};

Which was suggested in a different question that I cant find right now. In my case using every 100 made the document compilable even in pdflatex with an adjusted preamble, but distorted the plot too much. Compiling the document with pdflatex (with every 1) did not work for me, because the the system call from tikz was directed to pdfplots instead of lualatex despite the \tikzset statement.