I am loading quite a few data files via \pgfplotstableread, but do not further use them via pgfplots. I just kept them in the preamble and switched to plotting them in a standalone document. But now I've run my main document again, I've got:

PGFPlots: reading {xxxxxxxxx/xxxxxxxxx/xxxxxxxxx.csv}

Runaway definition?
->
! TeX capacity exceeded, sorry [main memory size=5000000].
\pgfplotsapplistXpushback@smallbufoverfl ...@toka
\the \t@pgfplots@tokb }\ex...
l.87 ...sv}{\xxxxxxxxxWordWordWordxxxxxxxx}
^^M

If you really absolutely need more capacity,
you can ask a wizard to enlarge me.


Kind of looks like I forgot to close some bracket somewhere. However on first check that does not seem to be the case. Is the loading of the data files already too much and should I outsource that?

# edit (2016-07-13)

In hindsight, I could have edited this into the post way earlier but unfortunately, as I am not that active anymore, I just caught wind of this question's popularity. So ultimately, I plotted the graphs and figures, respectively, one-by-one in an external file (with the help of the standalone class). For huge data, I used LuaLaTeX. To get a proper start for this, these might help:

• read the file just before its use and clear the table afterwards. it looks like there is not enough resources to keep them in the memory. – percusse Aug 9 '14 at 15:42
• @percusse Oh I didn't know about tat. – henry Aug 9 '14 at 15:57
• @percusse I suppose that is one solution. Considering the amount of data files I assembled them in specific sub-files anyway and keep them commented out if I do not need them. Rather safe than sorry, right. So if you like, you can post your comment as a reply and I'll mark it. – henry Aug 9 '14 at 16:01

Combining our chat and the comment to give some ideas though the real solution is mostly art instead of technique.

pgfplots doesn't create a miracle within TeX. It simply uses PGF for harvesting all the points(or data points) and then tells TikZ there you go these are the points these are the colors and linetypes, deal with it. When harvesting it collects it within TeX macros. That's what is defined by the TeX capacity. Now you can say why not using my gazillionabyte RAM memory as matplotlib, matlab etc. does. But it is what it is. TeX is old.

First tool is to enlarge the main memory used by TeX. Related questions are How to expand TeX's "main memory size"? (pgfplots memory overload) and especially Increase TeX capacity, as non-root

Couple of indirect tricks can still be pulled off though. For example,

1. you can read the data table, use it and then let the data macro to \relax. For example,

% \documentclass{....}
%.....
%\begin{document}
%.....
%Huge file
}\mytable
% Used in the drawing or typeset
%....
%....
%After the figure or table we overwrite the macro that holds the table
\let\mytable\relax
%....
%\end{document}


this still assumes that the table is small enough to be handled at one shot but two or more of them are not. Otherwise we are back to where we started.

2. You can read fewer variables by skipping every other 4-5 points if you have too much detail.

3. You can draw it in external programs and include the resulting plot by introducing to pgfplots. Section 4.3.7 in v1.10 manual explains this.

• Your post made me look around once agian. :) And then I also found \pgfplotstableclear{\TABLENAME} on page 46 of pgfplotstable.pdf. Your method of loading the table /data, I find it very clever. – henry Aug 17 '14 at 15:24
• @henry Its definition is \let#1=\relax ;) and just for the fun of it tex.stackexchange.com/q/141166/3235 – percusse Aug 17 '14 at 17:40