# Word count and 'TeX capacity exceeded'

I use this piece of code in the preamble, obtained from Environment that counts words inside:

\usepackage{xesearch}
\newcounter{words}
\newenvironment{assignment}{%
\setcounter{words}{0}
\SearchList!{wordcount}{\stepcounter{words}}
{a?,b?,c?,d?,e?,f?,g?,h?,i?,j?,k?,l?,m?,
n?,o?,p?,q?,r?,s?,t?,u?,v?,w?,x?,y?,z?}
\UndoBoundary{'}
\SearchOrder{p;}}{%
\StopSearching
\marginpar{\arabic{words} words}}


I use this in my main file that includes several chapters. If I include up to the end chapter 4 (33599 words) then the counting environment works fine. If I include the next chapter (~6K words) I get the TeX capacity exceeded error. Having tested and excluded all other options it is definitely down to the assignment environment reaching a buffer(?) limit at some point over 33.599K words. I am not sure how to create a MWE for this particular case without sharing my actual work which for obvious reasons I cannot do.

-
This may be just too hard for TeX, I don't know. Is using LuaLaTeX an option? If yes, you might want to look at this: gist.github.com/phi-gamma/2622252 (didn't test, just googled it). –  mbork Feb 23 '13 at 9:45
You can try whether you get the error with lots of automatically generated contents like this 100k long lipsum text. Then you post and example with a text PUT 100k OF TEXT HERE instead of the long text. –  tohecz Feb 23 '13 at 10:20
\blindtext also takes an optional repetition argument which could be made large enough to trigger the problem. –  Christian Feb 23 '13 at 10:42
Hi DGarside. A tip: if you copy and paste directly the link of the answer in your question the title will appear. –  Claudio Fiandrino Feb 23 '13 at 11:13
Can you add the complete error text? –  egreg Feb 23 '13 at 11:32

If I run the document below as posted I get a save stack error:

[30] [31] [32] [33] [34] [35] [36] [37]
! TeX capacity exceeded, sorry [save size=50000].


If I uncomment the two lines marked %%%% then it runs to the end

Output written on xew33.pdf (84 pages).


and reports 51720 words

\documentclass{article}

\usepackage{xesearch}
\newcounter{words}
\makeatletter

\newenvironment{assignment}{%
\endgroup
\setcounter{words}{0}%
\expandafter\ifx\csname wordcount@xs@searchlist\endcsname\relax
\let\wordcount@xs@searchlist\undefined
\SearchList!{wordcount}{\stepcounter{words}}
{a?,b?,c?,d?,e?,f?,g?,h?,i?,j?,k?,l?,m?,
n?,o?,p?,q?,r?,s?,t?,u?,v?,w?,x?,y?,z?}
\UndoBoundary{'}%
\SearchOrder{p;}%
\else
\expandafter\StartSearching
\fi}{%
\StopSearching
\marginpar{\arabic{words} words}%
\begingroup\expandafter\def\csname @currenvir\endcsname{assignment}%
}

\begin{document}

\def\a{one two three four five six seven eight nine ten. }
\def\b{red yellow green blue. }
\def\c{apple orange pear banana.}
\def\d{\a\a\b\b\c\a\b\c\c\b\b}
\def\e{\d\d\par\a\a\b\d\d\b\b\c\c\a\a\a\a\par}
\def\f{\e\e\a\a\b\b\e\c\d\d\a\c\b\a\e\e}

\begin{assignment}
\f\f\f\f\f\f\f\f\f\f
\f\f\f\f\f\f\f\f\f\f
\f\f\f\f\f\f\f\f\f\f
\end{assignment}

\begin{assignment}
one one two three
\end{assignment}

\begin{assignment}
one two three four five six
\end{assignment}

\end{document}


the document environment in LaTeX plays a similar trick to use the environment syntax but avoid making a group around the whole document.

-
This does calculate for size > 50000 but I do not seem able to have one assignment environment follow on immediately after another. The second environment registers as zero. Would the way around that be to create assignment_main and assignment_bibliography environments? –  DGarside Feb 28 '13 at 17:12
code updated to re-use the existing search structure seems to get the right answer on a couple of smaller lists at the end, see posted image –  David Carlisle Feb 28 '13 at 18:20