# Compiling a document with PGFplots processing only every x-th data point? [duplicate]

Compiling my document with many PGF Plots, each containing thousands of data points from a CSV file takes too long (minutes). I realize that it does not make sense to have this many data points -- were there, say, a tenth, it would be enough to recreate the same plot on paper.

One of my files, for example, consists of 216000 lines of such a format:
0.000000000000000000e+00 0.000000000000000000e+00 1.388888888888888888e-04 -2.182787284255027771e-11
Of course, that is far too much. I then read the file as:

\documentclass{scrartcl}

\usepackage{pgfplots}
\pgfplotsset{compat=1.5.1}

\begin{document}

\begin{tikzpicture}
\begin{loglogaxis}[
\end{loglogaxis}
\end{tikzpicture}

\end{document}


Is it possible for the package to take only every x-th line to speed up the compilation process? Or are there other ways to make it (significantly) faster?

I am running Ubuntu and a friend of mine recommended me to write an awk script that would go through a .csv file and retain only every x-th line. However, I have no clue how I would do that, as I have never used awk before. If this would turn out to be the option of choice, could someone help me setting up such a script?

-

## marked as duplicate by Count Zero, Marco Daniel, percusse, clemens, Torbjørn T.Aug 27 '12 at 6:12

We seem not to use the tag speed. We use compiling for questions concerning compiling time. –  yo' Aug 24 '12 at 12:24
Take a look at tex.stackexchange.com/questions/47787/…, for example –  Joseph Wright Aug 24 '12 at 12:48
Thanks, that was exactly what I was looking for. Feel free to close this one. Maybe we can make the other question more discoverable by adding tags or keywords or so? –  Ingo Aug 24 '12 at 13:33

pgfplots offers the filter each nth point=100.

Since coordinate filtering has different use-cases, you may want to set

each nth point=100, filter discard warning=false, unbounded coords=discard


this combination will not generate warnings for every discarded point. Furthermore, it will silently drop the 99 points - just as if you did not write them into the input stream at all. Note that the alternative choice unbounded coords=jump would explicitly interrupt your plot (which is not what you want here).

This approach has been report to work fairly well: in fact, the time was not much larger as if some external tool had been used to filter the data file (has been reported recently by some power user per mail).

-
I would suggest choosing a "nice" number for each nth point. Imagine that the orinigal sampling is on a grid 600x600 points. Then the points you leave there are aligned in one axis direction. If the number is e.g. 107, then the points will be spread better. –  yo' Aug 24 '12 at 21:56
Good idea. However, note that each nth point is applicable for one-dimensional data - it is unsuitable for grid data. –  Christian Feuersänger Aug 25 '12 at 9:25
Christian, thanks a lot for this first-hand answer. This is a really fantastic approach for my work-flow, since I can have the CSV file changing when running simulations and do not need to write a post-processing script whatsoever to edit the data afterwards. –  Ingo Aug 25 '12 at 10:35

Although the pgfplots filter each nth point might be a solution at some degree, it certainly isn't one that can be used in general. For example, choosing each nth point=100 might distort some areas of the plot that really need the extra points in order to be illustrated correctly. On the other hand setting each nth point to a lower number might not be "economic" in terms of the number of points being used.

In short, a cleverer downsampling method is needed. That method shoulb be able to dynamically adjust the number of points kept, depending on the complexity of the plot at each area, so that we have the best visual result using only the minimum required number of points (something like latex being able to automatically adjust each nth point=? several times during the "pass" of a diagram).

Take a look at this answer.

In case you have matlab or octave you can use the modified version of matlab2tikz provided there, which implements an effiecient downsampling algorithm.

On the other hand, if you use neither matlab nor octave, you can still translate that downsampling algorithm (it's in function pointReduction) to the language you want and use it to downsample your figures and then export them to CSV.

-
Note that @ChristianFeuersänger 's answer can be applied to different data regions. So one can manipulate the data in batches and save it as a new data file at each step. Via this, MATLAB functionality can be achieved which is possible through pgfplotstable. –  percusse Aug 24 '12 at 22:41
That's true, but choosing the regions manually for each one of the diagrams of an entire book (it is not guaranteed in general that the regions can be chosen in batch, that depends on the diagrams themselves), is just loosing precious time. The whole process of choosing the points to keep can be done automatically by a simple algorithm, in a much more efficient way. –  niels Aug 25 '12 at 8:00
What you propose is to use adaptive function compression - which is a reasonable idea, although with its own challenges and the requirement to use an external tool. Note that Ingo asked for a solution for "taking each nth line" explicitly. –  Christian Feuersänger Aug 25 '12 at 9:16
@ChristianFeuersänger, oh I merely think its a better idea to do the job outside latex with an external tool, the same one that was used to produce the "heavyweight" plots at first place. You can have far better results than just keeping every nth point. –  niels Aug 25 '12 at 9:28
@niels Agreed, thanks for the clarification. –  Christian Feuersänger Aug 25 '12 at 9:56