Compile extremely large file into pdf

I am using Django and it's templating system to dynamically generate Latex and then compiling it with pdflatex, which works great.

The problem is when the latex that gets generated is 30,000+ lines long and the compilation takes 2+ seconds, not actually outputting proper PDF files (i.e. tables are missing) and sometimes just crashing pdflatex altogether (error message is: run out of memory).

Is there a way I can improve the performance and/or avoid compilation crashes?

There is nothing wrong with my latex, because when I scale down the generated latex size everything works fine. It's only when the quantity increases that problems start to arise.

I have done some research and the majority of the answers seem to talk about tricks for fast compilation with plotting and compiling only parts of the document. These tactics don't apply to my situation, as I am not plotting (it's just massive chunks of text, table rows to be more specific) and I can't compile sections, because the majority of the content is dynamically generated each time.

Edit 1:

@David Carlisle I am using "tabulary" to generate the tables, as it's the only way I could get the columns to automatically adjust their widths as appropriately as possible. Yeah I would have though the memory footprint would be small still, but it seems to be massive.

@egreg and @David Carlisle For one where the Latex is around 30,000 lines long I don't get any error messages (log file or console), but I do get heaps of "Overfull/Underfull hbox" warnings (which I think are harmless).

As for a Latex file that causes pdflatex to crash, the error I get is:

Runaway definition?
->\global \let
! TeX capacity exceeded, sorry [main memory size=7000000].
\CT@setup ...\CT@color {\global \let \CT@do@color
\CT@@do@color \color }
l.94583 \end{tabulary}

If you really absolutely need more capacity,
you can ask a wizard to enlarge me.

Here is how much of TeX's memory you used:
1708 strings out of 495028
23123 string characters out of 6181498
7000001 words of memory out of 7000000
4949 multiletter control sequences out of 15000+600000
4782 words of font info for 16 fonts, out of 8000000 for 9000
14 hyphenation exceptions out of 8191
34i,8n,32p,232b,270s stack positions out of 5000i,500n,10000p,200000b,80000s
!  ==> Fatal error occurred, no output PDF file produced!


Edit 2:

I know essentially nothing about the internals of how Latex is implemented, so I provided my answer based on knowledge of computer science and software development. From my perspective using tables with this amount of sporadic data is susceptible to the poor performance (leaky abstraction) of the implementation. Hence why I have suggested to avoid using tables.

The reason why there are 10,000+ rows is because the text/data is for a computerised maintenance management system. The documents are not to be read by a wide audience and are not needed to be overly "pretty", they just need to be legible. Hence why I have suggested to avoid using tables.

Yes, every line will be of different shape, which might add to the explanation of why I am having performance problems.

• You don't say how you are producing tables of that length, longtable? If it just text there is very little memory overhead related to the size of the file as each page releases more or less all the tex memory as it is shipped out. – David Carlisle Jan 20 '15 at 0:19
• Welcome to TeX.SX! TeX is able to cope with much bigger files. If the output is not correct it may mean there are syntax errors in the generated file. – egreg Jan 20 '15 at 0:20
• what is the error message exactly, copied from the log file, from the line ! to ? – David Carlisle Jan 20 '15 at 0:21
• It's still not really possible to offer any advice as you have not shown any relevant input, you are running out of main mem, so you could increase that in texmf.cnf, or use lualatex which does dynamic memory allocation, but you haven;t really shown why you are using so much menory, tabulary needs to hold each table in memory so it can do trial run to determine widths but it should release that each time so a document with a few thousand tabulary tables should not be a problem. – David Carlisle Jan 20 '15 at 9:26
• You can increase TeX's main memory size. See, for example, here. – Eliad Jan 21 '15 at 4:50

TeX has no problem with memory limits and can print ten million lines or 217,392 pages on my laptop with the following code:

\documentclass{book}
\usepackage{luatexbase, luacode}

\begin{document}

\begin{luacode}
for i=1,10000000 do
tex.print('x','y','z', '\\par')
end
\end{luacode}

tests
\end{document}


Actually the limit of ten million lines was arbitrary and I am sure you can double this. You are only limited by the system's ability to handle large log files and pdfs. Try it first with one zero less in the loop.

As long as you issue paragraphs or page breaks TeX will cope with any amount of output, provided your system can handle the pdf output.

So my advice is to build the table line, by line and issue it as paragraphs (more or less that is how longtable works.

I have made the example as LuaLaTeX to help you with the loop. It will perform better with plain TeX and a simple \loop..\repeat macro.

• This looks promising, but could you please edit your answer to provide a small example of "So my advice is to build the table line, by line and issue it as paragraphs". – pleasedesktop Jan 22 '15 at 23:02