As package author of the memory-extensive package pgfplots, I have been asked to analyze some out-of-memory situation.
I could identify the "culprit"; it was some call to \pdfmdfivesum in which it crashed finally.
I am aware of some solutions how to enlarge or avoid memory limits, so please avoid suggestions how to avoid the problem.
My motivation here is: as a software engineer, I wished for some kind of "heap dump" in which I can inspect how much memory is currently being occupied by which "word" or whatever. This could hopefully allow optimizations and systematic improvements; i.e. by clearing unused registers or by restructuring macro expansion or whatever.
Do you know if human-readable heap dumps can be generated?
Here is some more insight into the problem that I tried to address.
Personally, I think that this section is more or less unrelated to the question above: I would really like to hear answers even if there is a simple solution to the problem at hand.
Anyway, if you see how to improve the situation, I would listen carefully.
The problem at hand was the main memory size. Apparently, matlab2tikz generated a 300k file containing a self-contained pgfplots figure along with (lots of!) data points. And the tikz external library attempted to load that file into main memory in order to compute its MD5 hash. This failed. Note that without the MD5 computation, the file could be processed. In fact, the tikz external lib uses \edef\pgfretval{\pdfmdfivesum{\meaning\tikzexternal@temp}} and the call to \meaning fails if \tikzexternal@temp contains these 300k words. I suppose that these words occur more than once in the main memory of TeX; and I would like to learn where and why. This is where I hoped to see a heap dump.
Runaway definition?
->
! TeX capacity exceeded, sorry [main memory size=3000000].
\tikzexternal@hashfct ...aning \tikzexternal@temp
}
l.105 \end{tikzpicture}
%
If you really absolutely need more capacity,
you can ask a wizard to enlarge me.
Here is how much of TeX's memory you used:
18462 strings out of 494578
804304 string characters out of 3169744
3000001 words of memory out of 3000000
21352 multiletter control sequences out of 15000+200000
\pdfmdfivesumhas an optional keywordfilethat allows the computation of MD5 for an external file. See also\pdf@filemdfivesumof packagepdftexcmdsthat adds support for LuaTeX.\begin{tikzpicture}).\meaninghere: it turns something like\relax, a single token, into 7 character tokens, exploding the size. It should cause decidedly less churn to write something like\pdfmdfivesum{\unexpanded\expandafter{\tikzexternal@temp}}to get an appropriate value. That way, one token stays one token, so the memory requirements "only" double. Plus what is taken from the string pool temporarily. With regard to debugging: you can set\tracingstatsto a positive value and look at the log file using almost-failing input sizes. Not exactly fabulous.\tracingstatsto 2 causes TeX to output memory usage at every\shipout. I just tried this with a MikTeX 2.9 system - it didn't work. Knuth mentions that some TeX Systems are optimized and ignore tracing commands - it could be MiKTeX ignores this specific command. (\tracingalldoes output lots of stuff, however, but the output would require some postprocessing).