# No room for a new \count with datatool

This question led to a new feature in a package:
datatool

## Update: 2013-1-19

The MWE worked fines with the new (v2.13) version of the datatool package using \DTLnewdbonloadfalse. But in my actual usage I am getting the message:

Runaway argument? \db@plist@elt@w \db@col@id@w 1\db@col@id@end@ \db@key@id@w Chapter\db@key@id@end@ \ETC.

! File ended while scanning use of \@dtl@get@keydata.

after approximately 16172 DB \DTLloaddb -- approximate as I am only counting the DB's that are repeatedly loaded.

Will try to create a MWE for this, but in case it is obvious to someone else what the problem is I thought I'd add this information while I try to get to the bottom of this.

## Background:

I am generating a document which tracks the progress on components of a project consisting of a large number of files. For each of these files, a tiny DB is generated and read via the datatool package. This DB contains a list of the related files and and where they were found, and what types of files they were.

Although this is not related to the MWE below, here is a small excerpt from the output to illustrate what I am actually trying to achieve:

from which I can tell that file

• 15-4-34 has some progress (shown in blue text)
• 15-4-35 and 15-4-36 has significant progress (highlighted).
• 15-4-37 has had no progress at all (shown in red)

Each of these are links to the files so I can just click on them to open them to view, and I can tell which team member has made progress. At the end of the document I produce a summary report so I know how far along I am into the project. There is a bunch of other info that is captured as well, but this should be sufficient to give an idea as to the actual use case.

The list of associated files is contained in two small CSV files that are generated at run time, for each component of the project. For instance, if a .tex file exists, I would then check that there was a corresponding PDF, and if any associated figures were completed, or if there was a \todonote in the .tex file which indicated that that this file was still not complete, etc..

## Problem:

While processing this document, I received the following error message:

No room for a new \count .

\et@xchk ...lse \errmessage {No room for a new #2}


As per egreg's answer at the first reference listed below, three are two potential causes for this:

1. Too many packages
2. A programming error, such as saying \newdimen inside the definition of a command.

After thoroughly searching my code for such programming errors, I determined that the culprit is the datatool package which declares a \newcount within \DTLnewdb:

\newcommand*{\DTLnewdb}[1]{%
\DTLifdbexists{#1}%
{%
\PackageError{datatool}{Database #1' already exists}{}%
}%
{%
\dtl@message{Creating database #1'}%
\expandafter\newtoks\csname dtldb@#1\endcsname
\expandafter\newtoks\csname dtlkeys@#1\endcsname{}%
\expandafter\newcount\csname dtlrows@#1\endcsname
\expandafter\newcount\csname dtlcols@#1\endcsname
}%
}


It appears to release these in \DTLdeletedb as follows:

    \expandafter\let\csname dtlrows@#1\endcsname\undefined
\expandafter\let\csname dtlcols@#1\endcsname\undefined


However, it appears that this is not really having the desired effect of returning the counter back to the available list of counters for subsequent use as the MWE illustrates.

## Notes:

• The MWE needs the file MyData.csv. However, I commented out \usepackage{filecontents} to prevent overwriting of this file to prevent accidental deletion.
• The output of the MWE below is not important, just there so a PDF gets created.

## Question:

• Is there a workaround for this, or perhaps a better way of using the datatool pacakge for such a thing?

## Code:

\documentclass{article}
\usepackage{datatool}
\usepackage{pgffor}

%\usepackage{filecontents}% Commented to prevent overwriting MyData.csv
\begin{filecontents*}{MyData.csv}
Directory, Color
../dirB,    red
../dirC,    yellow
\end{filecontents*}

\begin{document}

\foreach \x in {1,...,10000} {%             Read DB numerous times
\DTLifdbexists{MyDB}{\DTLdeletedb{MyDB}}{}%
}%

\par\noindent\DTLdisplaydb{MyDB}%           Some output to have a PDF

\foreach \x in {1,...,10000} {%             Read DB numerous times
\typeout{count=\x}%
\DTLifdbexists{MyDB}{\DTLdeletedb{MyDB}}{}%
}%

\end{document}

-
Could you explain what you are actually aiming to achieve: this is a good MWE but I'm not sure that loading 20k files is a good plan in TeX! – Joseph Wright Jan 9 at 19:04
@JosephWright: Added more details to show actual use case. – Peter Grill Jan 9 at 19:36
@barbarabeeton: Has been corrected. Thanks for pointing it out -- even though it does not effect the problem or solution I prefer to eliminate any cause of possible confusion. – Peter Grill Jan 21 at 0:33

Joseph has explained the reason why the problem arises. Here's a possible workaround. I define a set of macros that set up two pools of token registers and count registers, from which to choose new ones when \DTLdbnewdb is called and that are reinserted in the pool when \DTLdeletedb is issued.

One can increase the size of the pool, with the following macros it's limited to five databases open at the same time, which should be sufficient.

\usepackage{datatool}
\makeatletter

%%% Auxiliary macros
\def\pool@getnext#1{\expandafter\pool@getnext@aux\expandafter#1#1\@nil}
\def\pool@getnext@aux#1#2#3\@nil{\def#1{#3}\gdef\pool@temp{#2}}

\expandafter\def\expandafter\pool@strip@count\string\count#1\@nil{#1}
\expandafter\def\expandafter\pool@strip@toks\string\toks#1\@nil{#1}

\def\pool@count@list{}
\def\pool@toks@list{}
%%% Fill the lists; only 5 simultaneously open databases are allowed
\count@=\z@
\loop\ifnum\count@<10 % use 2n, where n is the maximum number of open db
\newcount\pool@temp
\newtoks\pool@temp
\repeat

%%% Allocation macros

\newcommand\pool@newcount[1]{%
\ifx\pool@count@list\@empty
\PackageError{datatool}{Count pool exhausted}{Too many open databases}
\else
\pool@getnewcount{#1}%
\fi}

\newcommand\pool@getnewcount[1]{%
\pool@getnext\pool@count@list
\global\expandafter\countdef\csname#1\endcsname=\pool@temp
\csname#1\endcsname=\z@
}
\newcommand\pool@newtoks[1]{%
\ifx\pool@toks@list\@empty
\PackageError{datatool}{Toks pool exhausted}{Too many open databases}
\else
\pool@getnewtoks{#1}%
\fi}

\newcommand\pool@getnewtoks[1]{%
\pool@getnext\pool@toks@list
\global\expandafter\toksdef\csname#1\endcsname=\pool@temp
\csname#1\endcsname={}%
}

\newcommand\pool@releasecount[1]{%
\expandafter\let\expandafter\pool@temp\csname#1\endcsname
\global\expandafter\let\csname#1\endcsname\@undefined}

\newcommand\pool@releasetoks[1]{%
\expandafter\let\expandafter\pool@temp\csname#1\endcsname
\global\expandafter\let\csname#1\endcsname\@undefined}

%%% Patch datatool

\renewcommand*{\DTLnewdb}[1]{%
\DTLifdbexists{#1}%
{%
\PackageError{datatool}{Database #1' already exists}{}%
}%
{%
\dtl@message{Creating database #1'}%
\pool@newtoks{dtldb@#1}%
\pool@newtoks{dtlkeys@#1}%
\pool@newcount{dtlrows@#1}%
\pool@newcount{dtlcols@#1}%
}%
}

\renewcommand*{\DTLdeletedb}[1]{%
\DTLifdbexists{#1}%
{%
{%
\expandafter\let\csname dtl@ci@#1@\@dtl@key\endcsname\undefined
}%
\pool@releasetoks{dtldb@#1}%
\pool@releasetoks{dtlkeys@#1}%
\pool@releasecount{dtlrows@#1}%
\pool@releasecount{dtlcols@#1}%
}%
{%
\PackageError{Can't delete database #1':
database doesn't exist}{}{}%
}%
}
\makeatother


The pool is initialized using the usual \newtoks and \newcount macros, so the registers will be reserved and not used by other packages.

Unfortunately, this seems to increase the execution time. I'll try and see whether this can be reduced.

A different way is to patch the macros of datatool so that they reuse the counters if a database is closed and its symbolic name is reused:

\makeatletter
\renewcommand*{\DTLnewdb}[1]{%
\DTLifdbexists{#1}%
{%
}%
{%
\dtl@message{Creating database #1'}%
\global\expandafter\let\csname dtldb@#1@active\endcsname\@empty
\@ifundefined{dtldb@#1@used}
{\expandafter\newtoks\csname dtldb@#1\endcsname
\expandafter\newtoks\csname dtlkeys@#1\endcsname
\expandafter\newcount\csname dtlrows@#1\endcsname
\expandafter\newcount\csname dtlcols@#1\endcsname}
{\csname dtldb@#1\endcsname{}\csname dtlkeys@#1\endcsname{}%
\csname dtlrows@#1\endcsname=\z@\csname dtlcols@#1\endcsname=\z@}
}%
}
\renewcommand*{\DTLdeletedb}[1]{%
\DTLifdbexists{#1}%
{%
{%
\expandafter\let\csname dtl@ci@#1@\@dtl@key\endcsname\@undefined
}%
\global\expandafter\let\csname dtldb@#1@used\endcsname\@empty
\global\expandafter\let\csname dtldb@#1@active\endcsname\relax
}%
{%
\PackageError{Can't delete database #1':
database doesn't exist}{}{}%
}%
}
\renewcommand{\DTLifdbexists}[3]{%
\@ifundefined{dtldb@#1@active}{#3}{#2}}
\makeatother


One must patch also \DTLifdbexists since the original relies on the fact that one of the token registers is allocated. So I define \dtldb@MyDB@active when a database is active and undefine it at deletion; when a symbolic name is "new", the registers are allocated and \dtldb@MyDB@used is defined, so that if the same name is reused, no new register is allocated and the existing ones are cleared.

-

TeX's registers can be addressed either by name or by number. To keep a track of the numbers allocated to names, it is necessary to have an allocation routine, which \newcount provides an interface for. Thus when you do

\newcount\foo


you define \foo to mean \count<number>, and mark <number> as 'taken'. If you subsequently do

\let\foo\undefined


you free \foo but do not mark the <number> as available.

The LaTeX allocation routine does not provide a mechanism to release register numbers so there is no

\deletecount


or similar. It would in principal be possible to write an allocation routine that allowed for this possibility, However, e-TeX allows us to use 32k registers, and TeX is a typesetting system not a general programming language. As such, it's never been necessary to write such an allocation routine.

Note that using e-TeX alone does not provide access to 32k registers of most types. You need an allocation routine that knows about them, most easily achieved using

\usepackage{etex}


That does not help here as there are 20k file load operations, and datatool uses two counts per table, so requires 40k of them!

-
 @NicolaTalbot: So then the two lines in \DTLdeletedb are not having the full desired effect. Would it make sense then to remove those two lines quoted in the question from \DTLdeletedb, and change \DTLnewdb to not declare a newcount if it already exsits and instead just reset the count to 0? – Peter Grill Jan 9 at 19:11 An allocation routine allowing for \releasecount would probably work by having a list of 'released' registers, and checking that before using the main pool. Certainly doable: not quite sure if this counts as part of the question! – Joseph Wright♦ Jan 9 at 19:21

I've just uploaded v2.13 to CTAN. It will probably take a day or so to propagate through the mirrors and distros, but once it's available you can use the code below. The best solution is to only define the database once (before the loop) and clear it each time round the loop. This means that you're not constantly allocating new registers. The new version allows you to adjust the behaviour of \DTLloaddb and \DTLloadrawdb so that they append to the named database rather than creating a new database. This is done via the conditional \ifDTLnewdbload and the commands that switch it on and off (\DTLnewdbloadtrue and \DTLnewdbloadfalse):

\documentclass{article}
\usepackage{datatool}
\usepackage{pgffor}

\usepackage{filecontents}% Commented to prevent overwriting MyData.csv
\begin{filecontents*}{MyData.csv}
Directory, Color
../dirB,    red
../dirC,    yellow
\end{filecontents*}

\begin{document}

\DTLnewdb{MyDB}

\foreach \x in {1,...,10000}{%             Read DB numerous times
\typeout{count=\x}%
\DTLcleardb{MyDB}% clear database
}%

\par\noindent\DTLdisplaydb{MyDB}%           Some output to have a PDF

\foreach \x in {1,...,10000}{%             Read DB numerous times
\typeout{count=\x}%
\DTLcleardb{MyDB}% clear database