I have several manuscripts each with their own bibliography (*.bib file). I'm now trying to combine these manuscripts into one document having one single bib file. My problem is that I have duplicate entries in these bib files that I would like to remove.

My OS is OS X Yosemite, so I have access to BibDesk if that's an option (I mention this because this came up in several Google searches prior to me asking this question). I've also tried to use bibtool to comment out duplicates. It works well, but only on entires that are exact duplicates. My problem is that for the same reference, I have different cite keys across my different bib files.

My question is: is there any way to clean up the merged bib file and either assign two cite keys to one reference---or---automatically detect duplicates (with different cite keys)?


  • I should mention that I've tried crossref, but since they have two different cite keys, the citation is still given a different number and a duplicate appears.
    – Amit
    Mar 14 '15 at 17:48
  • 1
    there's a tool bibclean that may be helpful; i haven't time now to check whether it handles this particular problem, but it can be found with google. even after you have combined duplicate entries, you will still be faced with the different keys in the different manuscripts. i don't know any way of merging those under one key except by hand. Mar 14 '15 at 17:53
  • @barbarabeeton: thanks, I'll check out bibclean
    – Amit
    Mar 14 '15 at 18:09
  • 1
    I use jabref to manage my bibtex files. It has a Find duplicates function that does exactly this. I find it to be a very efficient way to organise and search my now rather large main bibtex file, especially as it stills let me edit the "pure" bibtex source easily. There is a mac version (which you can manage with brew casks if you wish).
    – user30471
    Mar 14 '15 at 20:32
  • 1
    Is there any news here?
    – Johannes_B
    May 16 '16 at 14:25

Yes, both BibDesk and Jabref have internal duplicate finder tools. But none of them have advanced ways of cleaning up duplicates. While Jabref is still better in giving the user an option to Merge or select any instance of the duplicates, its algorithm of finding the duplicate is not transparent. It also often fails to detect entries with exactly the same title.

The problem is: both of the reference managers seem to detect duplicates only when the entireshave exactly the same values in all the required fields. I have about 4000 references. Both Jabref and Bibdesk detected a few references at first. After some cleaning up, then, they reported absence of duplicates. But I knew I have more than 500 duplicate entries in my library. I know this because I tested the library in Bookends; a reference manager which has more advanced duplicate finder tool.

To find a permanent solution, I have requested for more advanced duplicate finder tool in Jabref :http://discourse.jabref.org/t/more-control-on-the-duplicate-finder/120 (For the Java programmers, it would be great if you can offer a hand on the Jabref.)

In the meantime, I have asked in this website for a solution based on AppleScript (BibDesk accepts AppleScript). The user jacksr300 helped me with an AppleScript. I am now able to clean all my duplicates based on Title as well as CiteKey. Check out the answer here: https://stackoverflow.com/questions/39427926/find-and-remove-duplicates-in-a-bibtex-bibdesk-using-applescript/39439542?noredirect=1#comment66377804_39439542

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.