61

I was constructing a BibTeX file for my Thesis. At the middle of that I had copied that bib file to another document I had to produce, I had to add some new citations to the copy of the original.

Now I have 2 bib files that have a lot in common but differ in some citations. Is there a way to merge them? So that I have one single file with all the citations from both files but no duplicates?

| improve this question | |
  • This is a duplicate question. (I don't know how to mark it) And I have solved the problem with BibDesk for Mac OS. – Tiago Veloso Jun 6 '11 at 10:37
  • 2
    This question is not related to TeX. There are a lot of programs that can merge text files. In Linux you can try with Kdiff3 or meld. In Windows WinMerge is fine. The merge process might be automatic (if you just appended the citations), but it will propably need some manual handling (especially if you changed the data on some refs). – pmav99 Jun 6 '11 at 10:38
  • 12
    @pmav99 I think the TeX in BibTeX is enough to warrant a place here. Also WinMerge, Kdiff and all diff like tools work on a line-by-line basis, whereas BibTeX files work on entries and not lines. – Tiago Veloso Jun 6 '11 at 10:41
55

bibtool -s bibliography1.bib bibliography2.bib will merge two bib files, keeping duplicate entries. bibtool -s -d bibliography1.bib bibliography2.bib will merge two bib files, commenting out one of the duplicated entries (not sure which one). For more info, see documentation.

Bibtool is on CTAN. Not to be confused with bibtools which is also on CTAN and probably also has the capacity to do this sort of thing...

Bibtool can also be found in Ubuntu repositories. (I wasn't able to compile the one from CTAN)

| improve this answer | |
  • 2
    The current version of bibtool clings to duplicate entries. Removing them requires multiple steps: (a) pass -d on the command line, (b) create a resource file containing the entry print.deleted.entries = off, then (c) mention the resource file with -r on the command line. It does still work though, after all that. – Byron Hawkins Apr 26 '17 at 13:32
  • To obtain a bibtex file, running bibtool -s -d input1.bib input2.bib -o output.bib on Ubuntu was satisfactory for me. – Rasmus Mar 6 '18 at 12:51
  • 1
    To avoid changing capitalization and also the duplicated entries on the merged file we can use bibtool --preserve.key.case=on --print.deleted.entries=off -s -d input1.bib input2.bib -o output.bib – Leandro Nov 4 '18 at 5:39
47

JabRef is a bibliography manager that has tools for merging bib-files. Or rather, for importing one file into another: File --> Import into current database. Not sure if it checks for duplicates on merge, but there is a tool for checking for duplicates as well: Tools --> Scan database --> Find duplicates.

| improve this answer | |
  • 8
    +2 for Jabref. I normally don't fall for GUI programs. But Jabref did the merging quite well. Whats more, it arranged the new database alphabetically, something I had been wanting to do for a while. – cryptic0 Mar 26 '13 at 20:23
  • 5
    One can select "Deselect all duplicates" right before merging the two databases and thus obtain a set of unique references. – Marius Hofert Mar 24 '16 at 13:32
10

I think there is no easy way to do this. Though it is possible to write a perl script alike to use regex to do this. But the problem is

  1. Sometimes two entries are duplicates though they are not exactly the same, for example different capitalization in titles or extra {and }. So we have to workaround this by define some threshold for difference.

  2. Sometimes two entries from the same author of the same year with a little difference in the title are indeed two papers. This make the threshold in (1) hard to define.

So I suggest you can only do this by hand. A relative easier way is do it with a BibTeX editor, for example BibDesk on Mac OS X. Import both files into it, and it shall warn you when two entries have the same citekey. So for those entries with same citekeys, which are almost surely duplicates, you can clean them up easily. If I remember correctly BibDesk and other editors do have this functionality. After this, sort all entries by title, and spot further duplicates by your eyes.

| improve this answer | |
  • I had BibDesk, but I did not know of this feature. Sure made my life easier. – Tiago Veloso Jun 6 '11 at 10:44
  • 4
    I think it is a shame that this is the accepted answer given that there are two good ways to do this suggested in the other answers. – Seamus Jun 13 '13 at 20:56
5

Just to re-ensure Torbjorn's point, JabRef checks for duplicates when using File -->Import into current database. D indicating duplicate entries in current bib database

Entries marked with "D" are duplicates found by JabRef 2.10. You can then use "Deselect All Duplicate" button to get rid of the duplicates.

Thanks

| improve this answer | |
  • Tried to import my bibliography into Jabref and it became unresponsive for many hours, so I definitely do not recommend it. – Tarantula Jan 28 '19 at 1:51
3

Obviously you should not delete the original database files until you are certain you no longer need them. Add a couple of extra years after you are really sure, just to be on the safer side of certain.

Caveat emptor ...

Biber can do this. In tool mode, it removes entries with duplicate keys by default.

Note that this assumes that it is not necessary to merge entries, but only to ensure that exactly one copy of each entry with a unique bibkey ends up in the merged .bib file.

For example if you have something like

@article{bernard-travels,
  author = {Dickens, Bernard R.},
  year = 2003,
  pages = {3--4},
  volume = 4,
  journal = {Travelling}}

@book{bertha-stargazing,
  author = {Smithson, Bertha},
  year = 1987,
  publisher = {Great Discoveries},
  address = {Mars},
  title = {Stargazing for Gazelles}}

in one file and

@article{liz-tricks,
  author = {Davies, Elizabeth H.},
  year = 2010,
  pages = {31--43},
  volume = 56,
  journal = {Magicians' Monthly}}

@book{bertha-stargazing,
  author = {Smithson, Bertha},
  year = 1987,
  publisher = {Great Discoveries},
  address = {Mars},
  title = {Stargazing for Gazelles}}

then all will be well. But if the second file contains, for example,

@article{bernard-travels,
  author = {Dickens, Bernard R.},
  year = 2007,
  pages = {23--41},
  volume = 12,
  journal = {Travelling Again}}

@book{bertha-stargazing,
  author = {Smithson, Bertha},
  year = 1987,
  isbn = {1234567890123},
  address = {Mars},
  title = {Stargazing for Gazelles}}

then the result will probably not be what you want.

However, assuming unique bibkeys for unique entries and no need to merge individually partial entries, begin by combining the .bib files into one large .bib.

Suppose your large .bib is called big.bib. Then 

biber --tool --output-format bibtex --output-file merge.bib big.bib

will produce a combined .bib file, merge.bib, with entries sorted and de-duplicated.

biber --help

provides further details of the options available.

| improve this answer | |
  • 1
    Tried to use this approach, but biber removed fields and changed field names as well, so it made a complete mess, definitely not recommended as well. – Tarantula Jan 28 '19 at 1:52
1

Jabref is better at detecting the duplicates than BibDesk. But, since BibDesk has AppleScript support, some people have written tools for cleaning duplicates. I recently discovered this cool tool by Andreas Fischlin's tools: http://se-server.ethz.ch/staff/af/bibdesk/. The script that says Clean Duplicates can be modified. I modified the CiteKey to Title; and, able to find more than 400 duplicates that both BibDesk and Jabref missed. You can find the modified version here: https://www.dropbox.com/s/naegllc0yaril4q/Cleanup%20Duplicates.scpt?dl=0

But, be careful. It is dangerous. It can delete non-duplicate entries if they have the same Title by chance.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.