# How to use very large data sets with pgfplots w/o making pdf take a very long time to load?

I'm working with a very large data set - almost 120,000 observations - and I'm using pgfplots to make some plots. The process goes ok, especially with the use of the external library, but the pdf documents take a very long time to load. It seems as though adobe is individually drawing each data point.

Is there a better way to use pgfplots to make the plots easier to render?

My set up is:

\documentclass[12pt]{article}
\usepackage{tikz}
\usepackage{pgfplots}
\usepgfplotslibrary{external}
\tikzexternalize

\begin{document}

\begin{figure}[h]
\footnotesize
\centering
\caption{Publicly Available Num Tested, MEAP\_stacked Number Valid}
\begin{tikzpicture}
\begin{axis}[
only marks,
width=6.2in,
ylabel={Number Tested, Public Data},
xlabel={Number Valid, Our Data},
area legend,
legend style = {at={(axis cs:700,0)},anchor=south east},
xmin=-50,
ymin=-50]
\input{statistics/Num_Valid_Writing.tex}
\input{statistics/Num_Valid_Math.tex}
\input{statistics/Num_Valid_ELA.tex}
\input{statistics/Num_Valid_Science.tex}
\input{statistics/Num_Valid_SS.tex}
\end{axis}
\end{tikzpicture}
\end{figure}

\end{document}


I'm using pdfLaTeX and Ubuntu with TeXLive.

Thanks!

-
you can give PSTricks a try. It can read large data very quickly: PSTricks.tug.org – Herbert Jul 1 '11 at 5:30
@Herbert: The problem in this case seems to be that the resulting PDF takes a long time to display in the viewer, not the compilation time. Would the PDF resulting from a PSTricks approach load faster? – Jake Jul 1 '11 at 5:46
@Jake: when running the document with pst2pdf I can use a on the fly created png instead of a pdf – Herbert Jul 1 '11 at 5:52

This answer may need pgf CVS, I do not know.

The external lib can be customized to produce png on its own -- with correctly scaled images.

The pgfmanual for pgf (CVS?) describes the process in all detail in Section 34.7 Bitmap Graphics Export in its documentation of the external library.

The trick is to use


\tikzset{
% Defines a custom style which generates BOTH, .pdf and .png export
% but prefers the .png on inclusion.
%
% This style is not pre-defined, you may need to copy-paste and
png export/.style={
external/system call/.add={}{; convert -density 300 -transparent white "\image.pdf" "\image.png"},
%
/pgf/images/external info,
/pgf/images/include external/.code={%
\includegraphics
[width=\pgfexternalwidth,height=\pgfexternalheight]
{##1.png}%
},
},
%
png export,% ACTIVATE
}



There are three steps involved:

a) modification of the system call (no problem, should be possible with any version of the external lib)

b) the /pgf/images/external info key (which is probably pgf CVS, only) which generates TeX dimension information for every exported image and stores it appropriately.

c) the use of \pgfexternalwidth when loading the image (this is actually output of this 'external info' thing).

The \tikzset statement in my suggested solution defines a new style 'png export'. Rather than activating it globally as in my solution, you may just enable it for a single picture by


{% from here on, png export will be used....
\tikzset{png export}%
\begin{tikzpicture}
....
\end{tikzpicture}
}% here ends the png export

-
Let me stress that my solution is based on Imagemagick... if it is properly installed, it will work on both Windows and Linux. – Christian Feuersänger Jul 2 '11 at 18:51
I'll try to implement this the next time I have a chance... – PaulB Jul 12 '11 at 23:59
It just came to my mind that my solution has a flaw: it will convert the complete figure to png - including descriptions. In other words: the solution is probably inadequate for your needs. I agree with your comment below that it would be nice to automate the process of externalizing ONLY the graphics and to re-read that later-on. Hm. – Christian Feuersänger Jul 30 '11 at 18:51

There is a threshold at which a given plot is better to be shown as a raster image. This is an intrinsic limitation of vector formats.

The easiest solution can be to add one step to the externalization process such that the plots in external pdfs are converted into a raster format. For example by:

 convert -density 600 external_file.pdf -scale 800 external_file.png
mv external_file.pdf external_file-bak.pdf


The quality of the image can be controlled by the options in the Imagemagic 'convert' command.

For even better results it can be combined with the technique described in Section 4.2.8 "Using External Graphics as Plot Sources" described in the manual such that only the plot and not the axis are raster images; but that will require more work.

-
This will certainly do in a pinch - thanks. However, it would be really nice to automate the process. Also, it seems with .png I have to change my LaTeX file to set the height and width. I'll keep looking for ways to improve on this. – PaulB Jul 1 '11 at 14:36