# Plotting experimental data using data files with 5 thousand lines?

I want to illustrate various experimental data in my thesis e.g.:

• X-ray powder diffraction data (thousand lines)
• FTIR data (5 thousand lines)
• TG-/DTA data (5 thousand lines)

and I am wondering how to get a decent and convenient workflow with satisfying results.

I was playing with pgfplots and already be able to create a diagram for my XRD data (1k points) by expanding TeX memory size. But I failed to create a diagram for my FTIR experiments (5k points). Although I really want to present my diagrams in a consistent and qualitative manner it seems to me using pgfplots is somehow not really convenient for this kind of task?!

• In the case of pgfplots. How can I achieve to plot FTIR spectra with more than 5k points? Is it convenient to use pgfplots for this kind of task and if not...
• What would you recommend best for good workflow in combination with a considerable result for a lot of diagrams in my thesis?

My workflow so far: For preparation, I used qtiplot to normalize and transform plot data to show various plots in comparison in one diagram. After that I exported this manipulated data table to a .txt file.

\documentclass[a4paper,10pt]{article}
\usepackage[latin9]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{pgfplots}
%opening
\title{XRD pattern with pgfplots}
\author{Max Muster}

\begin{document}
\maketitle
\section{section with pgfplots-test}
\begin{tikzpicture}
\axis[
x dir       = reverse,
xlabel      = pgftemplate,
xmin        = 350,
xmax        = 4000,
yticklabels =
]
\datatable
\addplot[color=black,mark=none] table[y = 2nd plot] from \datatable ;
\addplot[color=black,mark=none] table[y = 3rd plot] from \datatable ;
\addplot[color=black,mark=none] table[y = 4th plot] from \datatable ;
\addplot[color=black,mark=none] table[y = 5th plot] from \datatable ;
\endaxis
\end{tikzpicture}

\end{document}

-
For serious statistics and good visualization you might want to consider the R language. With knitr you can keep your TeX source, statistical analyses and graphics generation all in the same file. – Ethan Bolker Jan 5 '14 at 13:17
Compiling with lualatex will allow you to use larger data files, but I think you're right that pgfplots (or actually TeX) is not ideal for this. – Torbjørn T. Jan 5 '14 at 13:28
Not really TeX related, but I'm intrigued by the resolution in your data. The FT-IR I tend to run has nothing like the number of data points you suggest, and the same was true last time I ran XRD. Do you really need all of that resolution across the full width? Have you considered only plotting part of the traces? (That can be done by pgfplots.) – Joseph Wright Jan 5 '14 at 13:29
I would second @JosephWright: if you have that many data points, any line-plot will result in relatively huge pdf files and it will take a long time to render -- and if the plots are "more or less smooth", you can reduce the samples without visible differences. – Christian Feuersänger Jan 5 '14 at 13:49
I don't use R myself but know that it's excellent for statistical analysis and graphing (my statistician son uses it all the time, with TeX). If I were starting from scratch knowing neither R nor pgfplots I would learn R, and knitr. If you're already almost there with pgfplots and don't imagine having to do much more than you have done, stick with it. – Ethan Bolker Jan 7 '14 at 0:40

This answer is referring to your wish to keep these graphs consistent with the rest of your thesis.

I believe that pgfplots can do this kind of stuff, although it takes longer than R.

However, you can easily gain lots of speed if you change your input method: replace \addplot table {\datatable}; by \addplot table {overview.txt}; and your time and probably mem usage will go down. At least the time consumption will reduce dramatically (see below for explanations)

I have just generated a couple of dummy plots to see if it works. I believe they resemble your use-case from a scalability point of view.

\documentclass{standalone}

\usepackage{pgfplots}

\pgfplotsset{compat=1.8}

\begin{document}

\begin{tikzpicture}
\begin{axis}
\end{axis}
\end{tikzpicture}
\end{document}


This attempt used

Here is how much of TeX's memory you used:
20614 strings out of 495035
531998 string characters out of 3781519
10114824 words of memory out of 15069104
23486 multiletter control sequences out of 15000+200000
3640 words of font info for 14 fonts, out of 8000000 for 9000
14 hyphenation exceptions out of 8191
62i,10n,76p,693b,1768s stack positions out of 30000i,500n,10000p,200000b,80000s


afterwards, I re-read the temporary files generated by gnuplot as follows:

\documentclass{standalone}

\usepackage{pgfplots}

\pgfplotsset{compat=1.8}

\begin{document}

\begin{tikzpicture}
\begin{axis}
\end{axis}
\end{tikzpicture}

\end{document}

Here is how much of TeX's memory you used:
20571 strings out of 495035
531135 string characters out of 3781519
10112441 words of memory out of 15066721
23441 multiletter control sequences out of 15000+200000
3640 words of font info for 14 fonts, out of 8000000 for 9000
14 hyphenation exceptions out of 8191
62i,10n,76p,694b,1762s stack positions out of 30000i,500n,10000p,200000b,80000s


It took a couple of seconds to generate these plots and my numbers give hints on the required memory settings. Note that I have used pdflatex here. lualatex will be even simpler as lualatex allocates memory dynamically and does not need cumbersome configuration changes.

If you need to do this all the time, you may want to consider alternative solutions.

If you need to do this occasionally (namely whenever you regenerate your data files), this is ok if you use the external library: in this case, the system will automatically compile the images into separate pdfs and will include the pdfs. In the optimal case, you use

\usetikzlibrary{external}
\tikzexternalize[mode=list and make]


as this will auto-detect if your input files change and will recompile if and only if needed. Use this if you are familiar with make. Let us know if you need help in this kind of setup.

The reason why \addplot table {<filename>} is much faster than \addplot table{<\loadedtable>}; is highly unexpected, and it resembles one of the most serious weaknesses in TeX: TeX has neither efficient arrays nor efficient lists. In fact, \addplot table {<\loadedtable>}; has quadratic runtime in the number of data points whereas \addplot table {<filename>}; is linear up to about 100000 data points. I am unsure but I believe memory also makes a difference (but only with respect to a factor).

-
Thank you, your suggestion helped. Right now I'm writing on a presentation with the beamer package containing only 2 FT-IR spectra. I'm really satisfied with the looks but compiling takes quite a long time and I'm concerned if it takes forever with my thesis. Furthermore, I'm concerned if I would ran into incompatabilites with LuaLaTeX and my preamble. Unfortunately I'm not familiar with make and I don't know how to use it. – user43602 Jan 6 '14 at 23:16
I see. I believe that you should really use the external lib; it is almost a "must do" in conjunction with huge docs + pgf/tikz/pgfplots. Without it, you will spend almost all of your time recompiling unchanged figures. Forget about my hint with [mode=list and make], simply write \tikzexternalize (without options), then execute with pdflatex -shell-escape. This will reduce you time considerably and is a best-practice. Make is capable of detecting changes in the included files automatically and supports parallel compilation, that's why I suggested it. But its optional. – Christian Feuersänger Jan 7 '14 at 7:24
Oh no. Today I tried your suggestion with my thesis. Unfortunately it did not work here. As far as I can see the created pdf's in the specified folder are okay but I get heaps of errors like ! Package pgf Error: No shape named inNote is known. Also, \tikzexternalize creates graphs of all my todonotes which results in errors too. pdf's in my TikzPictures still seem to be fine, though. – user43602 Jan 20 '14 at 17:13
@user43602 the todonote package does not work well with externalize, that is a known issue... work-arounds have been discussed on this site (somewhere). I do not know the shape message. Perhaps you should post some kind of minimal working example which reproduces the problem. – Christian Feuersänger Jan 20 '14 at 19:13
Well, the first thing I did was increasing TeX memory. But quickly I realized that this is not a solution. Because after adding some plots compiling takes ages. Then I removed all my todonotes commands but when I now comment out the todonotes package all my plots were not written correct anymore and I receive a message that my log file is greater than 3 MB. So I can't even remove todonotes anymore. When I use \tikzexternalize with todonotes now I get errors about misplaced } and \$. But I tested \tikzexternalize with another file on my system which went well. I don't know what to do. – user43602 Jan 25 '14 at 0:03