# How good a juggler is TeX?

I recently made an adaptation of pgfpages which allows one to gather in several typeset pages and then lay them out in some layout over several physical pages (the original pgfpages only allows one physical page). Thinking about other possible extensions, I started wondering just how many typeset pages TeX can hold in its memory at a time. In the comments to a Village's self-answer on How to sort the order of pages so that they can be printed in a folded book?, there's a hint that TeX can hold a large number of pages before actually shipping any out. And when I think about what little I know about a PDF document, it would appear that TeX has to process the entire document before it can actually write a single page to the output (I believe that the PDF has to say at the start how long it is).

PGFPages works by storing the pages in boxes, so this might be a bit different to how TeX stores actual pages before writing them to the file. If this is different, I'm more interested in how many boxes TeX can hold at a single time.

I realise that this also might be system-dependent, in which case I'd like to get some idea of how it is dependent, and what the "stock TeX" can do as I'd rather not modify the memory capabilities just for this juggling ability.

So how many boxes can TeX juggle before it runs the risk of dropping them?

-
by not modify memory abilities do you mean not recompile from source or not tinker with values in texmf.cnf? – David Carlisle Jul 26 '12 at 10:56
@DavidCarlisle Not tinker with values. I'm a long way from ever wanting to recompile from source! For myself, I'd tinker with texmf.cnf if I really wanted the extended memory, but if I were to distribute some code I wouldn't want to assume that the end-user was happy to do the same. – Loop Space Jul 26 '12 at 10:58
As I'm probably stretching the limits with DocScape beyond anything any normal TeX user would ever want, I'd be inclined to say "without limit". By modifying texmf.cnf you can blow up the size of the TeX process to several GB, so that you need at 64bit system to even handle it. Ridiculous amounts of stuff can be held in main memory then. I'm getting a memory usage of 253480166 words of memory out of 254931841. – Stephan Lehmke Jul 26 '12 at 12:58
More important question: How many TeXnicians can juggle? – Seamus Jul 26 '12 at 13:12
@StephanLehmke Would you consider posting some figures as an answer? Even though I'm more interested in the unmodified situation, knowing at what point you've found it necessary to modify texmf.cnf and how far you've pushed it makes it easier to see what "normal use" would count as. – Loop Space Jul 26 '12 at 13:13

Within limits you can adjust the size of allocated memory in texmf.cnf but to show the defaults using an unmodified default TL2012 (later versions should have similar values)

$pdftex "\def\n{100}\input box.tex" This is pdfTeX, Version 3.1415926-2.4-1.40.13 (TeX Live 2012) restricted \write18 enabled. entering extended mode (./box.tex ) No pages of output. Transcript written on box.log.  so that works, but increase by a factor of 10 and $ pdftex "\def\n{1000}\input box.tex"
This is pdfTeX, Version 3.1415926-2.4-1.40.13 (TeX Live 2012)
restricted \write18 enabled.
entering extended mode
(./box.tex
! TeX capacity exceeded, sorry [main memory size=3000000].
\body ->\hbox {}
\advance \a 1 \ifnum \a < 1000
\next ->\body
\let \next \iterate \else \let \next \relax \fi \next
\kbox ...{}\advance \a 1 \ifnum \a < 1000 \repeat
}
\body ->\hbox {\kbox
}\advance \a 1 \ifnum \a < \n
\next ->\body
\let \next \iterate \else \let \next \relax \fi \next
l.17 \repeat
}
!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on box.log.


The test file looks like:

\newcount\a

\def\kbox{{\a=0
\loop
\hbox{}%
\ifnum\a< 1000
\repeat}}

\setbox0\hbox{%
\a=0
\loop
\hbox{\kbox}%
\ifnum\a < \n
\repeat}

\bye


On the other hand, if you switch engines, it takes a while but:

$time luatex "\def\n{10000}\input box.tex" This is LuaTeX, Version beta-0.70.2-2012062812 (TeX Live 2012) restricted \write18 enabled. (./box.tex ) No pages of output. Transcript written on box.log. real 0m17.010s user 0m15.381s sys 0m1.575s  But if you push your luck too far: $ time luatex "\def\n{100000}\input box.tex"
This is LuaTeX, Version beta-0.70.2-2012062812 (TeX Live 2012)
restricted \write18 enabled.
(./box.tex
! TeX capacity exceeded, sorry [node memory size=197283195].
\body ->\hbox {}
\advance \a 1 \ifnum \a < 1000
\next ->\body
\let \next \iterate \else \let \next \relax \fi \next
\kbox ...{}\advance \a 1 \ifnum \a < 1000 \repeat
}
\body ->\hbox {\kbox
}\advance \a 1 \ifnum \a < \n
\next ->\body
\let \next \iterate \else \let \next \relax \fi \next
l.17 \repeat
}
!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on box.log.

real    0m56.810s
user    0m33.539s
sys     0m4.055s

-

When I was developing the booklet package, which puts pages into boxes and after a certain number of pages have been processed then outputs all the boxes in one go, I noticed that the number of boxes that TeX could handle depended on the content of the pages. The more complex the boxed pages the fewer boxes before memory overflow. I never quantified the relationship. So, following on from David's answer, mine is: it all depends.

-