Obviously you should not delete the original database files until you are certain you no longer need them. Add a couple of extra years after you are really sure, just to be on the safer side of certain.
Caveat emptor ...
Biber can do this. In tool
mode, it removes entries with duplicate keys by default.
Note that this assumes that it is not necessary to merge entries, but only to ensure that exactly one copy of each entry with a unique bibkey ends up in the merged .bib
file.
For example if you have something like
@article{bernard-travels,
author = {Dickens, Bernard R.},
year = 2003,
pages = {3--4},
volume = 4,
journal = {Travelling}}
@book{bertha-stargazing,
author = {Smithson, Bertha},
year = 1987,
publisher = {Great Discoveries},
address = {Mars},
title = {Stargazing for Gazelles}}
in one file and
@article{liz-tricks,
author = {Davies, Elizabeth H.},
year = 2010,
pages = {31--43},
volume = 56,
journal = {Magicians' Monthly}}
@book{bertha-stargazing,
author = {Smithson, Bertha},
year = 1987,
publisher = {Great Discoveries},
address = {Mars},
title = {Stargazing for Gazelles}}
then all will be well. But if the second file contains, for example,
@article{bernard-travels,
author = {Dickens, Bernard R.},
year = 2007,
pages = {23--41},
volume = 12,
journal = {Travelling Again}}
@book{bertha-stargazing,
author = {Smithson, Bertha},
year = 1987,
isbn = {1234567890123},
address = {Mars},
title = {Stargazing for Gazelles}}
then the result will probably not be what you want.
However, assuming unique bibkeys for unique entries and no need to merge individually partial entries, begin by combining the .bib
files into one large .bib
.
Suppose your large .bib
is called big.bib
. Then
biber --tool --output-format bibtex --output-file merge.bib big.bib
will produce a combined .bib
file, merge.bib
, with entries sorted and de-duplicated.
biber --help
provides further details of the options available.
diff
like tools work on a line-by-line basis, whereas BibTeX files work on entries and not lines.