8

I am writing my thesis and I copied and pasted some contents of previous .bib files. But maybe I used different labels for the same article title and maybe I cited these different labels in my thesis, so maybe my references can contain the same article twice. I have almost 190 references and I believe it will be hard to visually find repeated articles.

Is it possible to find in my bib file entries with the same title? I know bibtex looks for repeated labels. Is it possible to find repeated titles in my .bib file?

  • 5
    Are you using some sort of reference manager? Sort by title or author leads quite quickly to duplicate entries. – Johannes_B Mar 26 '16 at 12:50
8

You could use perl to go through the bib file, save all titles as a hash key with its line as the hash value, and then loop through it and print the title if its value has multiple entries. To do so, create a file with the following content, e.g. "finddupls.pl", change the bib file name, then execute perl finddupls.pl in your terminal:

#!perl
my %seen = ();

my $line = 0;
open my $B, 'file.bib';
while (<$B>) {
    $line++;
    # remove all non-alphanumeric characters, because bibtex could have " or { to encapsulate strings etc
    s/[^a-zA-Z0-9 _-]//ig; 
    # lower-case everything to be case-insensitive
    # pattern matches lines which start with title
    $seen{lc($1)} .= "$line," if /^\s*title\s*(.+)$/i;
}
close $B;

# loop through the title and count the number of lines found
foreach my $title (keys %seen) {
    # count number of elements seperated by comma
    my $num = $seen{$title} =~ tr/,//;
    print "title '$title' found $num times, lines: ".$seen{$title},"\n" if $num > 1;
}

# write sorted list into file
open my $S, '>sorted_titles.txt';
print $S join("\n", sort keys %seen);
close $S;

It returns directly in the terminal something like this:

title 'observation on soil moisture of irrigation cropland by cosmic-ray probe' found 2 times, lines: 99,1350,
title 'multiscale and multivariate evaluation of water fluxes and states over european river basins' found 2 times, lines: 199,1820,
title 'calibration of a non-invasive cosmic-ray probe for wide area snow water equivalent measurement' found 2 times, lines: 5,32,

And it additionally writes a file sorted_titles.txt listing all titles alphabetically ordered which you could go through and detect duplicates manually.

| improve this answer | |
4

If you can rely on the title field being identical, then a very simple:

grep -n 'title =' bibliography.bib | uniq -cdf 1

This will print only non-unique lines (-d) and the number of times they appear (-c) for the file bibliography.bib and the line number they appear (-n) in the bibliography file; the -f 1 tells uniq to ignore the first field, which would be this line number.

So if you get a line like:

     2 733:  title =    {Ethica Nicomachea},

You know that you have two appearances of title = {Ethica Nicomachea}, and the first of them appears on line 733 of your .bib file.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.