# Lualatex capacity exceeded with Gb's of free RAM

I'm trying to compile dense graphs generated my matlab2tikz (file size ~ 50Mb). I'm running lualatex and externalising with lualatex

\tikzset{external/system call={lualatex \tikzexternalcheckshellescape -halt-on-error -interaction=batchmode -jobname "\image" "\texsource"}}


With file sizes of ~ 5Mb things work ok, but the jump to 50Mb fails. When I look at the profile of the 5Mb run, the processes use a few 100Mb of RAM, and the externalised pdf is about 1Mb. This leads me to think that the jump to 50Mb should be fine. Moreover, the lualatex runtime is about linear in the number of data points according to the pgfplots manual, so memory consumption must be as well.

The lualatex error at the moment is that TeX capacity exceeded [token memory size=75830065]. This is strange since I have at least 12Gb RAM available when I run the process, so I don't think memory, if dynamically allocated, should cause this. Moreover, when I look at the profile it fails while it's only consuming a few 100Mb of RAM, so there's still a lot of room to grow.

1) Is there some other errant call to pdflatex, or something similar that isn't part of the dynamic memory allocation?

2) If not, why is lualatex failing? Is there some other parameter that I can expand?

Running TeX Live 2014 on Windows 8 with TeXstudio. Here's the end of the log file:

("../../Matlab/simulations/sigma f-poly k-3 i-0.0001 s-0.001 nt-1 nb-0.0
1 const-1 HDD-0 minT-40 ds-100"
Runaway text?
\par \begin {axis}[width=\threeW , height=\threeH , scale only axis,\ETC.
! TeX capacity exceeded, sorry [token memory size=75830065].
<argument> ...52 0.0961556466390988\\ 0.3653 0.096155646\8ETC

l.2002060 \end{axis}

If you really absolutely need more capacity,
you can ask a wizard to enlarge me.

Here is how much of LuaTeX's memory you used:
27544 strings out of 494693
100000,75830065 words of node,token memory allocated 15902 words of node memory still in use:
146 hlist, 17 vlist, 15 rule, 116 disc, 34 math, 400 glue, 167 kern, 47 penal
ty, 1534 glyph, 162 glue_spec, 4 action, 3 temp, 1 if_stack, 8 local_par, 3 dir,
avail lists: 2:32,3:415,4:706,5:138,6:1117,7:73,8:36,9:345,10:106
30092 multiletter control sequences out of 65536+600000
53 fonts using 1491859 bytes
62i,20n,83p,1229b,546s stack positions out of 5000i,500n,10000p,200000b,100000s
!  ==> Fatal error occurred, no output PDF file produced!


EDIT

As discussed in the comments, I tried preprocessing one of these files. Here's a MWE not of the memory error (because the file is now ~ 5Mb) but of the new error. Removing the externalisation fails here as well.

\documentclass[10pt,a4paper]{article}

\usepackage{pgfplots}
\usepackage{tikz}

\pgfplotsset{compat=1.11}
\usetikzlibrary{external}
\tikzset{external/system call={lualatex -shell-escape -halt-on-error -interaction=batchmode -jobname "\image" "\texsource"}}
\tikzexternalize[prefix=compiled/]

\begin{document}

\newlength{\threeW}
\newlength{\threeH}
\setlength{\threeW}{1.2in}
\setlength{\threeH}{1.8in}

\input{small.tex}

\end{document}


small.tex

EDIT 2

It looks like with large files lualatex failed when it saw $\text{z}$ in an axis label (that should of course be $z$, it was in the form as a result of the matlab2tikz process), and that's what caused the error, similar to Undefined control sequence \text when importing Matlab figures. That's strange because in small files this error isn't thrown. This is so absurd. I'll report back if this isn't actually the solution.

• Is it necessary to create those graphs with that big data? I mean, couldn't you reduce the points with barely no change in the “print”? – Manuel Nov 2 '14 at 9:08
• @Manuel, no it certainly isn't necessary. But even when I downsample with each nth point=1000 it doesn't help and I get the same memory error, which leads me to think it's not managing to even process the file, which is strange because 50Mb surely doesn't need more than 12 Gb to be processed. – bjorne Nov 2 '14 at 9:11
• @bjorne Make a downsample via matlab and not via pgfplots: 2,000,000 points is to big! – Paul Gaborit Nov 2 '14 at 9:18
• @PaulGaborit, I did that as well. If I preprocess the files and comment out or delete say 99% of the data lines it still fails (but not a memory issue) saying Extra } and then Missing }, which seems contradictory. When I look at the processed file, it's fine, all the brackets are in tact, I only delete data lines, so this doesn't make sense to me. – bjorne Nov 2 '14 at 9:29
• This happened to me today as well with a large tikz file exported from matlab. I have over 16GB of RAM available, and I can watch the ram usage and lualatex is definitely not allocating too much RAM here. Also, @bjorne, I don't have the $\text{z}$ problem in my tikz file...the token memory size reported is 63191721 – daaxix Jul 28 '15 at 20:21

This is not really an answer, since it does not solve the problem, nor add much clarification, but I think it is worth writing it and it is too long for a comment.

Something is wrong about your profile showing only 100Mb of used memory. Only the token memory used is much larger, as I'll show in short, and there is also other kind of memory in use (string memory, node memory, etc.)

I searched for the string "token memory size", which appears in your output, in luatex source code. It appears at textoken.w:

        smemory_word *new_fixmem;       /* the big dynamic storage area */
t = (fix_mem_max / 5);
new_fixmem =
fixmemcast(realloc
(fixmem, sizeof(smemory_word) * (fix_mem_max + t + 1)));
if (new_fixmem == NULL) {
runaway();          /* if memory is exhausted, display possible runaway text */
overflow("token memory size", fix_mem_max);
} else {
fixmem = new_fixmem;
}


So we can do some computations from here. The number which is output next to the message "token memory size" is the current value of variable fix_mem_max, which happens to be 75830065 in your case. This is the number of smemory_words reserved for the token memory. Each smemory_word is a structure (defined in textoken.h) with two fields, each one of the type halfword, which is defined elsewhere as an integer of 32 bits. So each "token" is 64bits, and thus the actual memory allocated for the token list is fix_mem_max*8 in bytes.

In your case this means 75830065*8=606640520 bytes, which is roughly 578Mb. Still far from your 12Gb, but much more than the 100Mb your profiler reported.

The error is triggered when TeX exhausts the available token memory (the 578Mb allocated in your case), and tries to expand it via realloc, and fails because the main system memory is exhausted too.

You can see in the above code that the new tried size is sizeof(smemory_word) * (fix_mem_max + t + 1)) where t has been computed previously as fix_mem_max/5, so in your case the numbers are: 8*(75830065+75830065/5+1) which gives 727968632 bytes (around 694Mb).

Although these amounts are far from the 12Gb of RAM you have, they are only the token memory. As said before, there are other kind of memories used by LuaTex, so the total can be well around 1Gb or more. Still, I don't know if this is enough to explain the problem or if there is some bug inside LuaTeX which prevents the actual use of all the available RAM. Nevertheless, these are big numbers.

• Thanks. Interesting. I've worked around the problem as mentioned, so it's less mission critical for me to investigate the source of this frustration. But this is an interesting start. Something is definitely fishy. I've upvoted. – bjorne Nov 3 '14 at 17:42
• If lualatex is built as a 32 bit process, it will only be able to access 2GB, 3GB, or 4GB of memory depending on how it was compiled (for windows at least, which is where my issue reared its head), this is consistent with the OP's issues. – daaxix Jul 28 '15 at 20:34