54

As package author of the memory-extensive package pgfplots, I have been asked to analyze some out-of-memory situation.

I could identify the "culprit"; it was some call to \pdfmdfivesum in which it crashed finally.

I am aware of some solutions how to enlarge or avoid memory limits, so please avoid suggestions how to avoid the problem.

My motivation here is: as a software engineer, I wished for some kind of "heap dump" in which I can inspect how much memory is currently being occupied by which "word" or whatever. This could hopefully allow optimizations and systematic improvements; i.e. by clearing unused registers or by restructuring macro expansion or whatever.

Do you know if human-readable heap dumps can be generated?


Here is some more insight into the problem that I tried to address.

Personally, I think that this section is more or less unrelated to the question above: I would really like to hear answers even if there is a simple solution to the problem at hand.

Anyway, if you see how to improve the situation, I would listen carefully.

The problem at hand was the main memory size. Apparently, matlab2tikz generated a 300k file containing a self-contained pgfplots figure along with (lots of!) data points. And the tikz external library attempted to load that file into main memory in order to compute its MD5 hash. This failed. Note that without the MD5 computation, the file could be processed. In fact, the tikz external lib uses \edef\pgfretval{\pdfmdfivesum{\meaning\tikzexternal@temp}} and the call to \meaning fails if \tikzexternal@temp contains these 300k words. I suppose that these words occur more than once in the main memory of TeX; and I would like to learn where and why. This is where I hoped to see a heap dump.

Runaway definition?
->
! TeX capacity exceeded, sorry [main memory size=3000000].
\tikzexternal@hashfct ...aning \tikzexternal@temp 
                                                  }
l.105 \end{tikzpicture}
                       %
If you really absolutely need more capacity,
you can ask a wizard to enlarge me.


Here is how much of TeX's memory you used:
 18462 strings out of 494578
 804304 string characters out of 3169744
 3000001 words of memory out of 3000000
 21352 multiletter control sequences out of 15000+200000
12
  • 3
    There is no need to load the external file into a macro. \pdfmdfivesum has an optional keyword file that allows the computation of MD5 for an external file. See also \pdf@filemdfivesum of package pdftexcmds that adds support for LuaTeX. Commented Apr 5, 2013 at 15:30
  • @HeikoOberdiek thanks for the note... I considered using that part. However, at that point where I have to compute the MD5, I have already read the file. I would have to generate a tmp file and reread that one (the procedure is triggered at \begin{tikzpicture}). Commented Apr 6, 2013 at 6:54
  • 2
    Would a LuaTeX solution be OK? I am not sure if it's possible, but in case you'd be OK with a LuaTeX answer I could put some blood into my brain and try to get it to work.
    – topskip
    Commented Apr 24, 2013 at 19:05
  • 8
    You should not be using \meaning here: it turns something like \relax, a single token, into 7 character tokens, exploding the size. It should cause decidedly less churn to write something like \pdfmdfivesum{\unexpanded\expandafter{\tikzexternal@temp}} to get an appropriate value. That way, one token stays one token, so the memory requirements "only" double. Plus what is taken from the string pool temporarily. With regard to debugging: you can set \tracingstats to a positive value and look at the log file using almost-failing input sizes. Not exactly fabulous.
    – user31698
    Commented Jun 3, 2013 at 19:38
  • 1
    WRT:your original comment. The closest thing to a "heap dump" I know are the "tracing" commands, see the TeX Book. The LaTeX Companion mentions a "trace package" that can turn tracing on and off. The TeX Book says setting \tracingstats to 2 causes TeX to output memory usage at every \shipout. I just tried this with a MikTeX 2.9 system - it didn't work. Knuth mentions that some TeX Systems are optimized and ignore tracing commands - it could be MiKTeX ignores this specific command. (\tracingall does output lots of stuff, however, but the output would require some postprocessing).
    – infowanna
    Commented Aug 20, 2014 at 7:26

1 Answer 1

2

TeX's core engine does not use dynamically allocated memory but preallocated memory pools. So there's nothing like a heap to get dumped.

The TeX base code does not provide a way to show the contents and pointers into the preallocated memory pools. So the answer to your question is: No, its impossible to generate a "usefull" TeX heap dump (which should be called memory pool dump). With LuaTeX it might be possible, as it does not use TeXs base code but a rewritten engine.

To extend TeX memory you can edit/customize the configuration file texmf.cnf and regenerate your formats in the tex -ini steps of your TeX Installation. That's why a wizard is needed here. But most times its better to rethink the tokenization of your TeX code.

8
  • I think you should have a look at the comments under the question too. This is not what is asked.
    – percusse
    Commented Jun 8, 2015 at 9:13
  • I think that exactly answers what is asked and why this question has so many votes. I don't think that many people have exactly the same problem, but ask themselves, if there is a method to debug allocations or to see the what is allocated in the memory pools aka. heap. And the answer is: No, there is no such method. I approved that by reading the original pascal source code of tex!
    – ikrabbe
    Commented Jun 8, 2015 at 12:18
  • 1
    I mean the author already is aware of such limitation of TeX and asks for any way to do it possibly with the new features of Lua or Xe LaTeX. It is not difficult to come to the conclusion you have given here with the question itself and the comments under it already. What you have briefly suggested here is already reported extensively in the authors package manual pgfplots. Thus this is rather trivial info for the task at hand. See the third paragraph especially.
    – percusse
    Commented Jun 8, 2015 at 12:23
  • 1
    Thanks for the answer! Given the age of this question and the total amount of answers, I suppose that your doubts regarding the availability of a positive answer are adequate. But why should a static memory pool as such be an argument against a human readable heap dump? After all, static pools are not so uncommon -- even java has it (although it rebuilds it from time to time). Commented Jun 8, 2015 at 15:07
  • 1
    It's not against a human readable heap dump, generally spoken, but there is no such method implemented. Actually the write statements into the memory pools are very basic and transparent. It might be possible to implement a primitive that just dumps the bare octals of all these pools and pointer lists into the pools. Together with a hash table of the tex primitives, such a dump should contain the complete state of the machine.
    – ikrabbe
    Commented Jun 9, 2015 at 7:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .