Large bibtex/biblatex database

Question

I'm not sure if it is a good idea to maintain a single large bibtex database or if I should split it into multiple smaller ones. I have compiled a very large India-specific database (over 100,000 records) from various sources.

So, what are the most important positives and negatives for one approach or the other one? Is there any difference in how bibtex and biblatex treat big data files? Are there any threats to be considered with either approach?

Answer 1

Personally, I have two different bib-files; one for publications by others and one for my own publication list. But even this is just a matter of convenience because the entry-groups differ between the two for a more convenient working with the bib-files themselves, not considering any LaTeX compilation.

My main bib-file is well in access of 1000 entries, and I never experienced any problems with any compiler (BibTeX, BibTeX8, BibLaTeX) so far. I have never heard of any instance where such problems occurred either. True, there is a limit for the number of possible bibliography entries (defined by bytes to be processed) in BibTeX, and also in BibTeX8 (altough in the latter you can expand that limit to such large values that I did not run into problems with my Diploma thesis which contained tons of citations). But this has to do with the number of citations in your document and not the file size of the bib-file, so if you run into this problem it would persist even if you would split your bib-file. As far as I know, BibLaTeX does not have such limits anymore at all.

Another thing to consider is how easy it is to merge entries from multiple bib-files into one bibliography. In BibLaTeX, where you can declare several source files easily and create the bibliography by an altogether different command, this is easily done. In BibTeX and BibTeX8 it is to my knowledge not so easy (if possible at all) to compile information from several bib-files into one reference list.

So I would say is it mainly a matter of personal taste, probably also depneding on how much you are using the bib-files directly---it can make sense to have several if the entries are very different in topic. Compilation will work with either version at least in BibLaTeX, but may require additional packages or merging of bib-files for BibTeX/BibTeX8.

Answer 2

As of the late 2010s, multiple BibTeX-supporting reference management software, which I have used and can accept BibTeX databases as input, would fail to load BibTeX databases with more than 5000 BibTeX entries.

While LaTeX can support multiple BibTeX databases/files in its \bibliography{} command, I want to simplify the process/heuristic for creating the BibTeX key that I assign for/to each BibTeX entry in my BibTeX database(s). E.g., to create my preferred BibTeX key, I use the author's last name, or the first co-author's last name, appended by the year. Smith2000 is an example. For publications that yield multiple BibTeX keys with the same preferred BibTeX key, I would append a letter (assigned in alphabetical order) to the preferred BibTeX key. This is based on the author-date method for referencing publications, or Harvard referencing style or Harvard citation style. It is a type of parenthetical referencing.

Else, it would cause confusion. E.g., if multiple BibTeX databases for a given LaTeX document use the same BibTeX key for different references, this would create problems when interpreting/processing the BibTeX databases/files. My guess is that the first occurring instance of the BibTeX key would have its entry used, while the rest are ignored. Also, if I am unable to track the error message to the subset of BibTeX databases/files with the problem, I would have problems addressing/fixing the problem.

Such problems can be avoided by using a script to create a set of all the BibTeX keys that are found in all the BibTeX databases. If any repeated instances of BibTeX keys are found, it can report an error/warning, so that we can be notified of the problem and fix it by changing the repeating/"offending" BibTeX keys to new BibTeX keys.

Reference for using multiple BibTeX databases/files:

https://en.wikibooks.org/wiki/LaTeX/Bibliography_Management#Getting_current_LaTeX_document_to_use_your_.bib_file

I do not have problems with typesetting LaTeX documents with 20,000 references, except for using the \cite{} LaTeX command to cite more than 20,000 references in a LaTeX document. I created a workaround for the problem by partitioning the list of BibTeX keys into multiple cite commands. I used a Python script to grab a list of all the BibTeX keys in my BibTeX database/file, and print the BibTeX keys in sets of 10 per \cite{} LaTeX command.

You can validate this by using the \nocite{*} LaTeX command to list all the BibTeX entries in the bibliography of the LaTeX document, without having to specify all of the BibTeX keys in the BibTeX database(s).

https://en.wikibooks.org/wiki/LaTeX/Bibliography_Management#No_cite

Using a single BibTeX database also makes it easier for me to share my BibTeX database with others. I do not need to keep track of where all the BibTeX databases/files are. I use a single BibTeX database for all LaTeX documents.

An advantage of sufficiently small BibTeX databases, of less than 1,000 BibTeX entries, multiple open-source, and monetarily free, reference management software would accept these BibTeX databases. Consequently, their BibTeX parsers would check for compliance with the BibTeX language. Else, you have to use the aforementioned \nocite{*} LaTeX command to typeset a simple LaTeX document to check if your BibTeX database(s) comply with the BibTeX language specification. You can also build your own parser, or script to check for common problems that you may make when creating and updating your BibTeX database(s).