# Plot large data

As you all know, TikZ has to face the limits in terms of memory, that is given by the TeX compiler.

For most of my plots, I have a maximum time step of around 1us and I record several waveforms over seconds, which leads to inherent memory issues.

I was thinking about using gnuplot and tikz, so that the axis would be rendered with tikz and the plot would be a raster image of the size I decide.

Any best practice?

(I'll post a minimal example by the end of the day or over the week-end)

Remark I think one of the way out would be to raster the data points and keep as vector drawing the legend, axes, etc. Ideally the raster should be made within the plot, so as to guarantee its correct size. I know its part of matlab2tikz 2.0 release

• Almost certainly you can't show this many points in a meaningful form. Some form of interpolation or range reduction is therefore realistic. – Joseph Wright Jun 5 '15 at 8:38
• You are talking about several million data points per waveform at this resolution. As mentioned you need to filter this somehow (for any plotting mechanism) as this resolution would only be resolve-able to the human eye on a plot covering at least 5 million A4 pages. – Aubrey Blumsohn Jun 5 '15 at 9:14
• Assuming your X data is at a constant interval, I would write a simple filter (outside of any plotting system) to remove every point which is less than some designated Y distance from the previous point and which is also not a maximum or minimum. – Aubrey Blumsohn Jun 5 '15 at 9:29
• All this is perfectly correct, but I'd like to invest as little time as possible in processing the data. That's why I proposed to plot the points with gnuplot as a raster image... – s__C Jun 5 '15 at 10:40
• Lualatex is less constrained in terms of memory. – JPi Jul 6 '16 at 12:08

When treating big vector data like this, I fear a lot the possibility of having (undetected) visual aliasing. Consider for example a sinusoidal signal with period 10 (arbitrary units), with a noise of period 0.11.

#! /usr/bin/env python3
#
import math
import numpy as np
import scipy as sp

t1 = np.arange(0.0, 100.0, 1e-3)
y1 = np.sin(2*math.pi*t1/10) + 0.2*np.sin(2*math.pi*t1/0.11)
raw = np.column_stack((t1, y1))
np.savetxt("rawdata.dat", raw)


The data is in file rawdata.dat, and you have 100000 points.

pgfplots will give you a "TeX capacity exceeded" but you can plot the thing with :

\documentclass[border=10pt]{standalone}
\usepackage{tikz}
\usepackage{pgfplots}\pgfplotsset{compat=1.13}
\usetikzlibrary{arrows.meta,positioning,calc}
\begin{document}
\begin{tikzpicture}[
]
\begin{axis}[
xmin=0, xmax=100,
ymin=-1.5, ymax=1.5,
axis x line = center,
axis y line = center,
axis line style = {thick, gray},
xlabel = {$x$},
% every axis x label/.append style = {below, gray},
ylabel = {$y$},
legend style = {nodes=right},
legend pos = north east,
clip mode = individual,
]
\addplot[blue]  table [x index=0, y index=1, each nth point={100}] {rawdata.dat};
\end{axis}
\end{tikzpicture}
\end{document}


using the each nth point feature. You'll obtain:

...which is utterly wrong. The noise seems to have a period 10 times the real one; the real one is visible in this gnuplot graph:

where you can see from where the error come. Any kind of subsampling must be executed with care to avoid this.

What I normally do is preprocess the data and find, for every slice of samples that will be drawn , the average, the maximum, and the minimum (add this piece of code to the above python script):

SAMPLE=100
np.savetxt("sampledata.dat", raw[0::SAMPLE, :])
#
# create the file with t, y, ymin, ymax
#
reducedlen = math.floor(len(t1)/SAMPLE)
reduced = np.zeros([reducedlen, 4])
for i in range(0, reducedlen):
j = i*SAMPLE
reduced[i, 0] = t1[j]
reduced[i, 1] = np.average(y1[j:j+SAMPLE])
reduced[i, 2] = np.min(y1[j:j+SAMPLE])
reduced[i, 3] = np.max(y1[j:j+SAMPLE])
np.savetxt("reduced.dat", reduced)


and then I abuse the error bars to use them to have a "noise band" around the averaged signal (btw: you should use a nicer anti-aliasing filter here. The average is just an example and can fail sometime). The code will be:

 \addplot[red,
error bars/.cd,
y dir=both,
y explicit,
% error bar style={line width=2pt,}, % if you need it!
error mark options={
red,
mark size=0pt,
}
]
table [x index=0, y index=1, header = false,
y error minus expr = \thisrowno{1}-\thisrowno{2},
y error plus expr = \thisrowno{3}-\thisrowno{1},
]{reduced.dat};


and the result is the following one — that may be not really nice, but it is safe.

BTW, the same diagram can be obtained also using fill between using the minimum and maximum, which is probably more logical:

\documentclass[border=10pt]{standalone}
\usepackage{tikz}
\usepackage{pgfplots}\pgfplotsset{compat=1.13}
\usetikzlibrary{arrows.meta,positioning,calc}
\usepgfplotslibrary{fillbetween}
\begin{document}
\begin{tikzpicture}[
]
\begin{axis}[
xmin=0, xmax=100,
ymin=-1.5, ymax=1.5,
axis x line = center,
axis y line = center,
axis line style = {thick, gray},
xlabel = {$x$},
% every axis x label/.append style = {below, gray},
ylabel = {$y$},
legend style = {nodes=right},
legend pos = north east,
clip mode = individual,
]
table [x index=0, y index=2, header=false]{reduced.dat};
table [x index=0, y index=3, header=false]{reduced.dat};
\addplot[red] fill between [of=minimum and maximum];
\end{axis}
\end{tikzpicture}
\end{document}


A raster image is a good idea (probably the best for such a use case, vector images would be too huge).

The key point is: can you export your raster image with a well-determined bounding box, i.e. one in which you always know the axis coordinate of the lower left corner and the axis coordinate of the upper right corner?

If so, you can make use of pgfplots and its \addplot graphics feature. The first example shown in the pgfplots manual is

\begin{tikzpicture}
\begin{axis}[enlargelimits=false,axis on top]
[xmin=-3,xmax=3,ymin=-3,ymax=3]
{external1};
\end{axis}
\end{tikzpicture}


which assumes that external1 is a (raster) image with tight bounding box and that the provided limits are those of the lower-left and upper-right corner, respectively.

Producing images with a well-determined bounding box is not so simple as it sounds on first glance, however: many programs generate artificial space, and the y range of your time series will probably influence the height of the exported image.

For a best-practise with "small overhead", you may need to write a code generator which generates both the external graphics and the associated axis limits. In the most simple case, the associated axis limits are fixed and the image will always fit.

References: pgfplots manual and the related question 3-dimensional histogram in pgfplots

• Not clear to me why a raster image is preferred over the infinitely simpler simple method of removing redundant data... – Aubrey Blumsohn Jun 5 '15 at 14:10
• @AubreyBlumsohn That depends on your data's smoothness. I have seen time series in which compression needs adaptive compression methods which are beyond "infinitely simple" (and might still need huge sample sizes). From my point of view, you can either invest smartness + research into a suitable compression in which case a pgfplots solution would be preferrable, or you can invest time into a solution with braindead raster images for the densely sampled time series. Feel free to elaborate on an answer regarding the smart compression approach (perhaps the OP can provide sample data). – Christian Feuersänger Jun 5 '15 at 14:47
• I don't think any sort of complex timeseries smoothing or anything is required. It is simply a case of geometry - is the data point far enough from the last data point on the size of anticipated paper to be resolvable by the human eye (or the printer). If you are starting with 5 million data points on a line on A4 paper that will usually get you down to a data size of a thousandth. Even if you have 500 very sharp spikes in the data (very un-smooth) you are not going to need 5 million datapoints. – Aubrey Blumsohn Jun 5 '15 at 16:45
• You have a point. Why don't you write it as separate answer? My answer gives hints regarding best-practises how to import raster images, yours might be the better answer to the question. – Christian Feuersänger Jun 6 '15 at 8:28
• @AubreyBlumsohn yes please write a separate answer. I usually run some Matlab code and then matlab2tikz. – s__C Jun 26 '15 at 9:11