# pgfplot: plotting a large dataset

I try to plot a dataset which is large using pgfplots. Since I'm aware of problems with large files, I used the external mode. I additionally increased main_memory from 3000000 to 6000000. It crashes nevertheless, saying that pdflatex exceeded main memory. But if I look at the memory consumption of my system during compilation, I can't see any increase in memory usage. I couldn't find any explanation what main_memory takes, is it byte or Kbyte? If I increase it at a larger value, fmtutil-sys --all will fail and pdflatex won't work anymore. Can I get around this somehow?

\documentclass{article}

\usepackage{pgfplots}
\usepackage{pgfplotstable}

\usepgfplotslibrary{external}
\tikzexternalize

\begin{document}

\begin{tikzpicture}
\begin{axis}
\addplot table[x expr=\coordindex, y index=0] {largefile};
\end{axis}
\end{tikzpicture}

\end{document}


I also tried to compile with lualatex (just replaced pdflatex with lualatex) but this also seems to call pdflatex? At least I got this error message (same for lualatex and pdflatex):

! Package tikz Error: Sorry, the system call 'pdflatex -halt-on-error -interact
ion=batchmode -jobname "report-figure0" "\def\tikzexternalrealjob{report}\input
{report}"' did NOT result in a usable output file 'report-figure0' (expected on
e of .pdf:.jpg:.jpeg:.png:). Please verify that you have enabled system calls.
For pdflatex, this is 'pdflatex -shell-escape'. Sometimes it is also named 'wri
te 18' or something like that. Or maybe the command simply failed? Error messag
es can be found in 'report-figure0.log'. If you continue now, I'll try to types
et the picture.


Excerpt from report-figure0.log:

! TeX capacity exceeded, sorry [main memory size=6000000].
\pgfplotsapplistXXpushback@smallbufoverfl ...toka
\the \t@pgfplots@tokb \the...
l.13 ... expr=\coordindex, y index=0] {largefile};
^^M


largefile is 5.7 Mb is size and has 593932 datapoints. I barely dare to say that I also have a file of 163 Mb in size with 18928305 datapoints. I thought this is no problem because gnuplot can handle those files fast without problems.

head of largefile:

9409252
17298051
21351017
24466010
26952485
29389696
31442872
33345635
35029538
36710432


I'm using texlive2012:

$pdflatex This is pdfTeX, Version 3.1415926-2.4-1.40.13 (TeX Live 2012) restricted \write18 enabled$ lualatex
This is LuaTeX, Version beta-0.70.2-2012052410 (TeX Live 2012)
restricted \write18 enabled.

-
can you provide the data file for download? –  Herbert Sep 14 '12 at 8:27
If you want to use lualatex with the external library you must set the external/system call key so that lualatex is called in the subprocesses. See the documentation of tikz/pgf. –  Ulrike Fischer Sep 14 '12 at 8:38
I hope you realize that it does not really make sense to handle files of this size without a filtering process. –  Paul Gaborit Sep 14 '12 at 9:18
@Herbert: I added the link to the dataset in the description. –  Oliver Sep 14 '12 at 9:33

That's one very densely sampled dataset! As Paul Gaborit pointed out in the comment, you'll want to downsample this first, especially if the data is this smooth. Plotting every single data point will blow up the file size and decrease rendering performance without adding any value to the plot.

You can use gnuplot within PGFPlots to downsample the data. If you only plot every 1000th point, the plot is indistinguishable from the full data, but you can stick with pdflatex instead of having to use lualatex.

\documentclass{article}

\usepackage{pgfplots}
\usepackage{pgfplotstable}

\usepgfplotslibrary{external}
\tikzexternalize

\begin{document}

\begin{tikzpicture}
\begin{axis}
\addplot [no markers] gnuplot [raw gnuplot] {
plot "largefile.csv" using ($0*1000):1 every 1000; %$0 is the dummy column for the coordinate index
};
\end{axis}
\end{tikzpicture}

\end{document}


If you need non-uniform sampling, you can just concatenate several plot commands:

\documentclass{article}

\usepackage{pgfplots}
\usepackage{pgfplotstable}

\usepgfplotslibrary{external}
\tikzexternalize

\begin{document}

\begin{tikzpicture}
\begin{axis}
\addplot [no markers] gnuplot [raw gnuplot] {
plot "largefile.csv" using ($0):1 every 1::1::300; plot "largefile.csv" using ($0*100+300):1 every 100::300::6000;
plot "largefile.csv" using (\$0*1000+6000):1 every 1000::6000
};
\end{axis}
\end{tikzpicture}

\end{document}

-
Thanks for that explanation! It is not exactly the answer to my question but a practical workaround that should do the trick. Especially the non-uniform plotting is good because the first entries are more important. –  Oliver Sep 14 '12 at 12:00
I don't have my datapoints in a CSV file since I am using matlab2tikz. Am I still able to use gnuplot to reduce the number of datapoints in some manner? –  WG- Dec 13 '13 at 12:56

you can use this simple sed one liner to reduce the data, eg for every 200th value

sed -n '1~200p' < large.data > large200.data


then it can easily be plotted.

An alternative for the first 200 and then every 1000th:

sed '1,200!d' < large.data > large1000.data
sed -n '201~1000p' < large.data >> large1000.data

-
Thanks for your suggestions, Herbert. I prefer Jake's solution since it keeps the data unmodified and I don't have another file (I have already a lot of data files, so I'm happy to have not more files that will confuse me :-)). Additionally I can do adjustments on the fly by just changing the tex code. –  Oliver Sep 14 '12 at 14:47