10

As per my earlier question TeX capacity exceeded (save size), I am generating a fairly large index of .txt, .tex, and .pdf files. This process has worked for a long time, and the size of the index is not the issue. I have added logic to insert additional links in the index and now am about to exceed the string characters limit:

Here is how much of TeX's memory you used:
     134371 strings out of 493308
     9799992 string characters out of 9887816 
     4157963 words of memory out of 5000000
     59945 multiletter control sequences out of 15000+600000
     167812 words of font info for 205 fonts, out of 8000000 for 9000
     959 hyphenation exceptions out of 8191
     87i,12n,83p,10548b,16447s stack positions out of 5000i,500n,10000p,200000b,80000s

I have exceeded this several times in the past few months, and have optimized much of my code, but am running out of ideas as to where to further optimize. So, am trying to resolve this before I get stuck again, which seems imminent.

As a benchmark, processing a single file results in:

Here is how much of TeX's memory you used:
     52846 strings out of 493308
     1137494 string characters out of 9887816
     1523413 words of memory out of 5000000
     54776 multiletter control sequences out of 15000+600000
     167812 words of font info for 205 fonts, out of 8000000 for 9000
     959 hyphenation exceptions out of 8191
     87i,11n,83p,10548b,1042s stack positions out of 5000i,500n,10000p,200000b,80000s

Notes:

  • The string characters sometimes decreases slightly for no apparent reason. Since my indexing is processing a list of files, I can not see how adding another file to this list can possibly reduce the number of string characters required, but I have seen this happen.

  • I have already increased the pool_size in my /usr/local/texlive/2013/texmf.cnf:

    pool_size=10000000
    
  • Apologies for not providing a MWE. This process access my file system and to duplicate it would require setting up a bunch of files.

Questions:

  1. What are string characters?
  2. What kind of constructs should be avoided, and what kind are preferred to minimize using string characters ?
  3. Is there a way that I can see the number of string characters used in the middle of the run. If I could get this report before and after a macro invocation, I could possibly narrow down the main culprit.
  4. What does it take to increase this limit much further?

A solution to Question 4 would be ideal. I don't mind things taking longer. Currently takes about 40 minutes to generate the 600 page index, but even an overnight solution would be acceptable so lots of room left in terms of run time.

References:

  • this is from "deep memory" (mine), and may be faulty, so if someone like @DavidCarlisle corrects it, i won't be insulted. if memory serves, a file name is entered in the string pool when the file is opened. if nothing else has been posted to the pool before that file is closed, the file name is removed. if the name of the next file is shorter, then the size of the pool can decrease. (tex can't "jump over" entries in the pool, but it's able to remember the immediately previous entry.) you can keep the pool size down by giving files short names. – barbara beeton Jan 10 '14 at 20:43
7

It's the pool size: the letters used in command names, mostly. You can increase it in texmf.cnf usually (unless you reach the compiler limits)

% Max number of characters in all strings, including all error messages,
% help texts, font names, control sequences.  These values apply to TeX.
pool_size = 6250000

a simple test file (plain pdftex)

\tracingstats1
aaa

\def\a{aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa}
\def\b{\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a}
\count0=0
\loop
\advance\count0 1
\expandafter\def\csname \b \romannumeral\count0 \endcsname{}
\iftrue\repeat

This will keep making new csnames until it fills the pool, but it seems I can keep pushing it up in texmf.cnf

the effective pool seems to increase up to

pool_size = 100000000

which produces an error of

! TeX capacity exceeded, sorry [pool size=39921916].
 39921697 string characters out of 39921916

If I try to increase it further in texmf.cnf I get no warning but the reported size doesn't change. But that is approx 4 times bigger than the value you report.

  • the pool also contains file names. if there are a lot of files, and the names are long, that can eat up the pool pretty fast. (see "tex: the program", starting with "module" 511.) – barbara beeton Jan 10 '14 at 20:57
  • I already have pool_size=10000000, and seems that the actual value is less than this. Any way to increase it further? If it is being used to store the file names as per @barbarabeeton's comment , then how can I reuse the same string space -- I don't need the files open once I process them. – Peter Grill Jan 10 '14 at 22:02
  • @PeterGrill -- the comment on pool_size in module 11 of "tex: the program" says that this size "must exceed string_vacancies [set there to 8000] by the total length of TeX's own strings, which is currently about 23000". given that i'm looking at an early printing, those numbers may have changed, but i conjecture that the difference between what is reported in your log and your initial setting includes the length of "tex's own strings". – barbara beeton Jan 10 '14 at 22:29
  • @PeterGrill are you generating a lot (thousands) of long command names? (and can you double that value in the nf file?) – David Carlisle Jan 10 '14 at 22:32
  • @PeterGrill see update with test file – David Carlisle Jan 10 '14 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.