# Plotting floating point data with pgfplots

Is there a binary file structure supported by pgfplots to allow floating point numbers to be loaded and plotted? I understand the time taken to convert floating point data to a text file is small, but for large amounts of data (such as an animation), the time to process the file can be quite high. It seems illogical to convert the data from a floating point format, only for pgfplots to load and convert it back. Additionally, could somebody please explain how coordinates are translated to PDF data - I believe PDF objects can contain 32 bit floating point numbers. Is it therefore possible to place a floating point coordinate from a binary file directly into a PDF object, thereby avoiding 2 (or 3?) stages of conversion (and indeed retaining lossless coordinates)?

I realise that the structure of data presented in a binary file is less apparent than that of a plain text file, but would be able to manipulate the binary (or binaries) accordingly. I've included a basic example to try and make things clearer.

\documentclass{article}
\usepackage{filecontents,pgfplotstable}
\pgfplotstableset{create on use/x/.style={create col/copy column from table={x.csv}{0}}}
\begin{filecontents}{x.csv}
1.1
1.2
1.3
\end{filecontents}
\begin{filecontents}{y1.csv}
2.5
3.2
2.9
\end{filecontents}
\begin{filecontents}{y2.csv}
2.6
3.0
3.1
\end{filecontents}
\begin{document}
\begin{axis}
\end{axis}}

\begin{axis}
\end{axis}}
\end{document}


Instead of the contents of 'x.csv', 'y1.csv' and 'y2.csv' being those provided above, they would be binaries with the following hexdumps:

x.csv

40200000 404ccccd 4039999a


y1.csv

40266666 40400000 40466666


y2.csv

3f8ccccd 3f99999a 3fa66666

-
Try TeX internal accuracy without the fpu library. You'll appreciate the logic. If you are animating by reading data from a file on-the-fly, you are doing something very inefficient. –  percusse Mar 28 '13 at 17:32
Sorry to have not been clear - I'm hoping to do this in LaTeX - I've modified my question to provide an example which will hopefully demonstrate what I'm trying to do. I realise repeated file access isn't ideal, but my reasoning behind reading on-the-fly was that the next file could be read into memory to replace the previous, rather than using a huge amount of memory. –  mynoduesp Mar 28 '13 at 19:07

You stated two questions and I will address them briefly:

1 Can pgfplots read binary representations of floating point numbers?

In short: no.

2 How are coordinates mapped into a PDF?

This is a quite complex operation: first all coordinates are collected in order to compute axis limits. This is the "survey phase". Afterwards, all coordinates are mapped into the 32 bit fixed point number range of TeX/PDF/PS. These numbers are written in plain text into the output file (after applying any transformations like scaling or stretching).

Aside from these answers to your questions, I see that you are actually searching for something else for which these answers are merely some "sub-product". You appear to be wondering how to optimize something; apparently loading huge bulks of data files of animations or perhaps processing by means of pgfplots.

I would agree that animations might involve adequate data file formats. But if a binary "CSV" is the right one appears to be questionable. And pgfplots as tool to read "huge animation data files" appears to be as questionable. Do you want to import AVIs?

If you want to reduce the time that pgfplots ponders on data in general, you should pose a feature request or bug report. Note that it is entirely unclear of whether number parsing is a bottle neck at all (a pity that there are no powerful profilers). In fact, I would expect bottlenecks somewhere else.

If you want to improve the quality by reducing numeric operations, you may want to write some low-level driver file which writes stuff directly to a PDF. There are some operations in PDF which actually expect binary data, but they expect mapped integers rather than floating point numbers. I would expect that such an approach produces exactly the same quality as that produced of pgfplots (and would hope for a bug report if not).

I hope these thoughts help to improve the search for an answer and to clarify the use-case(s) that you have in mind.

-
Thanks for your detailed response. I'm producing vector animations and haven't found a better workflow. Other methods seem convoluted and produce inferior results. I was speculating improvements that would be possible using floating point binary data, since the workflow I'm currently using involves a conversion - there seems to be lots of switching of number representations. For this reason, I considered reading binary data directly to be a beneficial feature, but can see other operations performed by pgfplots may limit potential speed improvements. –  mynoduesp Apr 25 '13 at 9:21
Unfortunately, I'm not knowledgeable enough to write a low-level driver, but if you could steer me in the right direction and give some advice, I'll give it a go. A preliminary, crude idea is to pre-process binary data to mapped integer coordinates with the minimum at 0, maximum at 2^32-1, other values transformed as appropriate and maximum and minimum x and y values passed as 4 decimal coordinates. The integers can then be written directly to the PDF object. I'd appreciate your thoughts on this - it may be infeasible, but I'd be happy to refine it and do my best to contribute to it. –  mynoduesp Apr 25 '13 at 10:36
pgfplots actually uses the very same binary mapping (0 : 2^32-1) to implement its smooth shadings. Perhaps its output can be used as inspiration. Anyway; I think you should clearly define what output you'd what to see. "vector animations" can be all- or nothing. Perhaps it would be a good idea to post a new question containing a sketch of what you have right now, explain what is missing/sub-optimal, and ask for advice. Perhaps someone can show tweaks, short-cuts, or alternative packages. –  Christian Feuersänger Apr 26 '13 at 18:43

1 Can pgfplots read binary representations of floating point numbers?

In short: no.

I know the Topic is old, but for guys like me who have the same problem with the floating point format: pgfplots works now fine with floating point data (e.g. in csv format). Refer to the pgfplots manual page 21.

Cheers

-
binary is the problem there. –  percusse Sep 27 '13 at 18:18