# memory problems after migration to texlive 2016 and forest 2.0

I have an 800+ page book and it compiles fine with xelatex and forest 1.05 under texlive 2015. I am migrating to texlive 2016 since my translator works with forest 2.0. Now I get:

! TeX capacity exceeded, sorry [pool size=6143546].
\safeiterate@4 ...@countc y\endcsname )\endcsname
\forest@inpath      \forest@tem...
l.770 }


If I remove the forest code, I get the error later. If I tex smaller portions everything is fine. So I guess it really is a memory problem.

Question: Is there anything I can do apart from modifying some tex config files? The code is supposed to run for several users and having to modify memory variables would be bad.

Sorry, there is no minimal example here, but I could make the code available on github or somewhere else.

Edit: OK. It must be forest 2.0 since when I load forest 1.05 while running texlive 2016, everything is fine.

• Imho it would be the best to contact the forest author. Perhaps he is using to many local definitions with different names (I managed to overflow the pool size once with to many named coordinates tex.stackexchange.com/questions/316999/…). – Ulrike Fischer Jan 26 '17 at 18:45
• I'll gladly take a look if you make the code available. – Sašo Živanović Jan 26 '17 at 19:56
• Thanks! Here is the full code: github.com/langsci/25 I will sent details per mail. – Stefan Müller Jan 26 '17 at 21:02
• Don't use german.sty. It is really outdated. – Ulrike Fischer Jan 27 '17 at 11:19
• You need to remove \forestset{.style={for tree={parent anchor=south, child anchor=north,align=center,base=top}}}. It is not supported and no longer works. See p. 99 of the current manual: The unintended and undocumented way to specify defaults using \forestset{.style={...}} (see question Making a certain tree style the default for forest at TEX SE) does not work anymore. (Actually, it has never truly worked, and that’s why it has not compat key.) Use default preamble. – cfr Jan 27 '17 at 16:30

This was indeed a forest issue, but, surprisingly enough, has been present since the very first version of the package. Stefan was simply the first to create such a long, treefull document ... and the switch from v1 to v2 did nothing but push the issue over the edge.

Before explaining what went wrong: I have just posted the fixed version (v2.1.4) to CTAN.

The problem was precisely what Ulrike Fischer mentioned in a comment above. Forest's packing algorithm needs to (temporarily) store some information about coordinates (and pairs of coordinates). Furthermore, given a coordinate (or a pair), it needs to retrieve the information about it fast. The obvious solution is to store the information in a dictionary (associative array), with coordinates being the lookup key, so using TeX's control sequences seemed a perfect idea and I naively did (essentially copy-pasting from my proof-of-concept python implementation):

\csdef{forest@(\the\pgf@x,\the\pgf@y)}{...}


and even

\csdef{forest@(\the\pgf@xa,\the\pgf@ya)--(\the\pgf@xb,\the\pgf@yb)}{...}


not realizing that although the definitions are local, the entries will remain in TeX's hash table forever. This approach easily used up a few kilobytes of string pool space per tree!

v2.1.4 reimplements the offending dictionaries by storing all the info in a single toks register, whose contents look like this (shown only for the first of the above problems):

...(x1,y1){...}(x2,y2){...}...


It is easy to search for a specific coordinate in such a structure (although slower than in \csname approach):

\def\forest@breakpath@getfromtoks#1#2#3#4{%
% #1=cache toks register, #2=receiving cs, (#3,#4)=point;
% we rely on the fact that the point we're looking up should always be present
\def\forest@breakpath@getfromtoks@##1(#3,#4)##2##3\forest@END{##2}%
\edef#2{\expandafter\forest@breakpath@getfromtoks@\the#1\forest@END}%


(Many packages use such a system, see e.g.\ PGFs \pgfutil@in@.)

The new system is about 10% slower, but: in Stefan's 800+ page book, where version v2.1.3 exceeded the 6 million char string pool limit, v2.1.4 (and all the other many loaded packages) uses up a meager 2 million. With respect to memory consumption by the packing algorithm, the document length doesn't really matter anymore.

Stefan, thanks for finding this and bearing with me for the past week! (Hint: looking at the packing algorithm anew after these couple of few years, I believe it could also be made much faster!)

• Thanks!!!!!!!!! (Had to add some explanation marks to make the comment long enough =:-) – Stefan Müller Feb 4 '17 at 7:06