The following document uses the datatool package, and is slow to compile, though the resulting PDF file is only 3 pages long. On my computer it takes of the order of 17 sec. I'm hoping for suggestions to speed things up.
Clearly most of the time is spent in datatool's database manipulations. The single line
\dtlsort{RowID}{docDB}{\dtlletterindexcompare}
currently takes approximately half the time of the compile, but it is not clear to me why.
UPDATE: Thus far there are two comments, basically saying - for the data processing steps (sorting and merging), use tools more suitable for the job. However, I'm unclear why using a fast sort algorithm (if that is part of the issue) is a problem. I found What are the Easiest/Cleanest way to create arrays for illustrating quicksort with tikz? which talks about quicksort in TeX.
Also, I'm automatically creating the table in question from the references in text based on @egreg's answer: Automatically creating a table from datatool using references in the text, so I can't easily separate the data processing step from the LaTeX file if I want to continue doing that. I suppose I could merge and sort the data before passing it to datatool...
I compile this with pdflatex filename
.
######################
dto.tex
######################
\documentclass{letter}
\usepackage{datatool}
\usepackage{datagidx}
\usepackage{longtable}
\usepackage{lipsum}
\usepackage{array}
\usepackage{etoolbox}
\usepackage{url}
\setlength{\oddsidemargin}{-0.10in}
\setlength{\evensidemargin}{-0.10in}
\setlength{\topmargin}{-0.3in}
\setlength{\headsep}{0.2in}
\textheight=8.8in
\textwidth=6.7in
\newcounter{tabenum}\setcounter{tabenum}{0}
\newcommand{\colhead}[1]{\multicolumn{1}{>{\bfseries}l}{#1}}
\newcommand{\nextnuml}[1]{\refstepcounter{tabenum}\thetabenum.\label{#1}}
\makeatletter
\let\oldref\ref
\def\ref#1{%
\immediate\write\@auxout{%
\string\gappto\string\ReferencedIDs{#1,}%
}%
\oldref{#1}%
}
\def\ReferencedIDs{}
\makeatother
\usepackage{fouriernc}
\address{Some Address\\ Some Place\\Email: foo@bar.com}
\signature{(Somebody)}
\newcommand*{\checkmissing}[1]{\DTLifnull{#1}{}{#1}}
\newcommand{\PrintDocTableParekh}[3][]{%
% #1 = list of rowIDs
% #2 = database to search
% #3 =caption
\begin{longtable}{r l p{1.5in} c c p{2.5in}}
\caption{#3}\\
& \colhead{Date} & \colhead{Filename} & \colhead{From} & \colhead{To} & \colhead{Subject}\\\hline\endhead
\DTLforeach
[%
\(%
\equal{#1}{}\AND\DTLisSubString{\ReferencedIDs}{\RowID}%
\)%
\OR\(
\DTLisSubString{#1}{\RowID}\AND\DTLisSubString{\ReferencedIDs}{\RowID}%
\)%
]
{#2}{%
\RowID=RowID,%
\Date=Date,%
\Filename=Filename,%
\From=From,%
\To=To,%
\Subject=Subject%
}{%
\nextnuml{\RowID} & \Date & {\bfseries\expandafter\url\expandafter{\Filename} } & \checkmissing{\From} & \checkmissing{\To} & \Subject \\
}%
\end{longtable}
}%
\makeatletter
\newcommand{\AddToTable}[2]{%
\DTLforeach*{#2}{\newDBRowID=RowID,\newDBDate=Date,\newDBFilename=Filename,\newDBFrom=From,\newDBTo=To,\newDBSubject=Subject}{
\DTLnewrow{#1}
{\let\DTLnewdbentry\relax % Avoid expansion of \DTLnewdbentry when using \protected@xdef below
\protected@xdef\insertnewdbentry{%
\DTLnewdbentry{#1}{RowID}{\newDBRowID}%
\DTLnewdbentry{#1}{Date}{\newDBDate}%
\DTLnewdbentry{#1}{Filename}{\newDBFilename}%
\DTLnewdbentry{#1}{From}{\newDBFrom}%
\DTLnewdbentry{#1}{To}{\newDBTo}%
\DTLnewdbentry{#1}{Subject}{\newDBSubject}%
}}\insertnewdbentry
}}
\makeatother
\begin{filecontents*}{documents.csv}
1952.02.19,19 Feb 1952,something.txt,subject
1994.03.26,26 Mar 1994,something.txt,subject
2010.11.04,04 Nov 2010,something.txt,subject
smyt.bs.ie,Mar 2004-2013 (incl.),something.txt,subject
ct.bs.ie,Mar 2004-2013 (incl.),something.txt,subject
cf.bs.ie,Mar 2004-2013 (incl.),something.txt,subject
ismail.bs.ie,Mar 2004-2013 (incl.),something.txt,subject
mh.bs.ie,Mar 2004-2013 (incl.),something.txt,subject
smyt.it,2007-2013,something.txt,subject
ismail.it,2008-2013,something.txt,subject
2013.10.05.powai,05 Oct 2013,something.txt,subject
2013.10.05.kanjur,05 Oct 2013,something.txt,subject
2013.10.28.zhm,28 Oct 2013,something.txt,subject
2014.02.26,26 Feb 2014,something.txt,subject
familytree,26 Feb 2014,something.txt,subject
trusts,26 Feb 2014,something.txt,subject
\end{filecontents*}
\begin{filecontents*}{correspondence.csv}
1998.09.14,14 Sep 1998,something.txt,FROM,TO,,subject
1998.09.16,16 Sep 1998,something.txt,FROM,TO,,subject
2001.05.23,23 May 2001,something.txt,FROM,TO,,subject
2007.06.10,10 Jun 2007,something.txt,FROM,TO,,subject
2011.05.09,09 May 2011,something.txt,FROM,TO,Y,subject
2011.09.09.slm,09 Sep 2011,something.txt,FROM,TO,Y,subject
2011.09.26.fm,26 Sep 2011,something.txt,FROM,TO,Y,subject
2011.09.26.slm,26 Sep 2011,something.txt,FROM,TO,Y,subject
2011.09.26.zhm,26 Sep 2011,something.txt,FROM,TO,,subject
2011.11.21.slm,21 Nov 2011,something.txt,FROM,TO,Y,subject
2011.11.21.fm,21 Nov 2011,something.txt,FROM,TO,Y,subject
2011.11.22,22 Nov 2011,something.txt,FROM,TO,Y,subject
2011.11.23,23 Nov 2011,something.txt,FROM,TO,Y,subject
2012.01.25,25 Jan 2012 (?),something.txt,FROM,TO,,subject
2012.12.07,07 Dec 2012,something.txt,FROM,TO,,subject
2013.01.14,14 Jan 2013,something.txt,FROM,TO,,subject
2013.08.29,29 Aug 2013,something.txt,FROM,TO,Y,subject
2013.09.09.fm,09 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.09.rm,09 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.10.slm,10 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.10.fm,10 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.10.slm2,10 Sep 2013,something.txt,FROM,TO,,subject
2013.09.11.slm1,11 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.11.fm,11 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.11.slm2,11 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.16,16 Sep 2013,something.txt,FROM,TO,,subject
2013.09.17,17 Sep 2013,something.txt,FROM,TO,,subject
2013.09.23,23 Sep 2013,something.txt,FROM,TO,,subject
2013.10.06,06 Oct 2013,something.txt,FROM,TO,,subject
2013.10.14.kma,14 Oct 2013,something.txt,FROM,TO,,subject
2013.10.14.ks,14 Oct 2013,something.txt,FROM,TO,,subject
2013.10.16,16 Oct 2013,something.txt,FROM,TO,,subject
2013.10.17,17 Oct 2013,something.txt,FROM,TO,,subject
2013.10.18,18 Oct 2013,something.txt,FROM,TO,,subject
2013.10.22,22 Oct 2013,something.txt,FROM,TO,,subject
2013.10.23,23 Oct 2013,something.txt,FROM,TO,,subject
2013.10.24,24 Oct 2013,something.txt,FROM,TO,,subject
2013.10.25.zhm,25 Oct 2013,something.txt,FROM,TO,,subject
2013.10.25.fm1,25 Oct 2013,something.txt,FROM,TO,Y,subject
2013.10.25.rm1,25 Oct 2013,something.txt,FROM,TO,Y,subject
2013.10.25.fm2,25 Oct 2013,something.txt,FROM,TO,Y,subject
2013.10.25.rm2,25 Oct 2013,something.txt,FROM,TO,Y,subject
2013.10.28,28 Oct 2013,something.txt,FROM,TO,,subject
2013.10.30,30 Oct 2013,something.txt,FROM,TO,,subject
2013.11.11,11 Nov 2013,something.txt,FROM,TO,,subject
2013.11.12,12 Nov 2013,something.txt,FROM,TO,,subject
2013.11.13,13 Nov 2013,something.txt,FROM,TO,,subject
2013.11.25,25 Nov 2013,something.txt,FROM,TO,,subject
2013.12.05.ta,05 Dec 2013,something.txt,FROM,TO,,subject
2013.12.05.slm,05 Dec 2013,something.txt,FROM,TO,,subject
2014.01.15,15 Jan 2014,something.txt,FROM,TO,,subject
2014.02.24,24 Feb 2014,something.txt,FROM,TO,,subject
2014.04.02,02 Apr 2014,something.txt,FROM,TO,,subject
2014.04.10,10 Apr 2014,something.txt,FROM,TO,,subject
2014.04.15,15 Apr 2014,something.txt,FROM,TO,,subject
2014.07.18.tm,18 Jul 2014,something.txt,FROM,TO,,subject
2014.07.18.ta,18 Jul 2014,something.txt,FROM,TO,,subject
2014.07.24,24 Jul 2014,something.txt,FROM,TO,,subject
2014.09.05.tm,05 Sep 2014,something.txt,FROM,TO,,subject
2014.09.05.ta,05 Sep 2014,something.txt,FROM,TO,,subject
2014.09.05.slm,05 Sep 2014,something.txt,FROM,TO,,subject
2014.10.17,17 Oct 2014,something.txt,FROM,TO,,subject
2014.11.12.tm,12 Nov 2014,something.txt,FROM,TO,,subject
2014.11.14,14 Nov 2014,something.txt,FROM,TO,,subject
2014.11.17,17 Nov 2014,something.txt,FROM,TO,,subject
2015.01.06,06 Jan 2015,something.txt,FROM,TO,,subject
\end{filecontents*}
\begin{document}
\def\today{9th January, 2014}
\begin{letter}{
Someone\\
Somewhere\\
Subject: Some stuff
}
\opening{Dear Someone}
\lipsum[1]. Here is a ref - [\ref{2014.11.14}].
\lipsum[2]. Here is a ref - [\ref{2014.11.14}].
\lipsum[3] Here is a ref - [\ref{2014.11.17}].
\lipsum[4]
\lipsum[5] Here is a ref - [\ref{2014.11.14}].
\lipsum[6] Here is a ref - [\ref{2014.07.18.tm}].
\lipsum[7]
\lipsum[8] Here is a ref - [\ref{2014.11.12.tm}].
\lipsum[9]. See references
~[\ref{2013.10.05.kanjur}], [\ref{2013.10.05.powai}].
[\ref{2013.10.06}], [\ref{2013.10.28.zhm}],
[\ref{2013.10.17}], [\ref{2013.10.22}],
\lipsum[10]. See references
[\ref{2013.10.14.ks}], [\ref{2013.10.06}]
[\ref{2013.10.17}], [\ref{2013.10.22}], [\ref{2013.10.24}],
\lipsum[10]. See references
[\ref{2013.10.28}], [\ref{2013.10.30}]
\lipsum[10]. See references [\ref{2013.11.11}]
\lipsum[11]. See references [\ref{2014.09.05.tm}]
\lipsum[12]. See references
[\ref{2014.07.18.tm}] [\ref{2014.11.12.tm}]
\lipsum[13]. See references
[\ref{2014.11.17}], [\ref{2015.01.06}], [\ref{2015.01.06}]
\closing{Yours Sincerely,}
\DTLloaddb[noheader,keys={RowID,Date,Filename,Subject}]{docDB}{documents.csv}
\newtermaddfield[docDB]{From}{From}{}
\newtermaddfield[docDB]{To}{To}{}
\DTLloaddb[noheader,keys={RowID,Date,Filename,From,To,Email,Subject}]{corrDB}{correspondence.csv}
\AddToTable{docDB}{corrDB}
\dtlsort{RowID}{docDB}{\dtlletterindexcompare}
\PrintDocTableParekh{docDB}{Documents}
\end{letter}
\end{document}
n^2
, and also, is very weak in merging databases, even sorted ones; on such operation, it truly reach the quadratic complexity.datatool
in any way, it is surely advisable to pre-sort your 'data' before feeding it to your.tex
file. There are many tools....