Take the 2-minute tour ×
TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. It's 100% free, no registration required.

I have a large .bib file which is in a mess. In particular:

  1. Entries are not in any particular order.
  2. Some entries may be duplicates (albeit with different keys).
  3. Some entries may not be used in the relevant book.

(1 is an issue because I'm working through the .bib file fixing up items one by one and examining the corresponding changes in the book; at the moment I'm having to dart back and forth a lot.

3 is an issue because I need to check the correctness of each entry manually, which can take a while; thus unused entries mean a lot of wasted time.)

Is there a way of taking a .bib file and cleaning it up? In particular, I'd like to

a) create a .bib file which is sorted by author, and

b) delete or, better, comment out unused entries.

share|improve this question
8  
Open your .bib file with JabRef and it will complain about duplicates if you search for them from some menu option that I don't remember at the moment. Then also you can sort it as you wish by clicking the columns etc. –  percusse Oct 12 '12 at 11:44
3  
I think the checkcites script might help, at least for listing the unused entries. The script analyzes both .aux and .bib files; in order to list only unused references in your .bib file, use $ checkcites --unused mydoc.aux in the command line. :) Disclaimer: I'm one of the authors of the script. –  Paulo Cereda Oct 12 '12 at 11:57
    
I would use some program to manage your .bib entries. BibDesk om mac or see tex.stackexchange.com/questions/9454/… –  Hans-Peter E. Kristiansen Oct 12 '12 at 13:19
    
JabRef works beautifully -- thank you. If you turn it into an answer, I will accept it. (I didn't try checkcites as JabRef made it easy enough to detect unused entries -- but thanks for the suggestion.) –  Mohan Oct 12 '12 at 13:48
    
Mohan, just for your information: When you start a comment with "AT username", for example AT percusse, that user will get notified of your post. It is not necessary to use AT OriginalPoster, here AT Mohan, because the original poster of a question or of an answer gets notified automatically. (Replace AT by @ - only one @ allowed per post...) –  Stephen Oct 12 '12 at 14:19

6 Answers 6

up vote 26 down vote accepted

I would recommend bibexport script. It creates a new .bib file that includes only the references you cite in the .tex file, and cleans them up. I use it for submissions to journals, when I want to send only the relevant references rather than my databases.


Also JabRef has some duplicate search and resolve capabilities which can be accessed via the menu items shown (for v2.8) below:

enter image description here

share|improve this answer
    
How nice! I didn't know about bibexport, thanks! :) –  Paulo Cereda Oct 13 '12 at 1:50
4  
I've added the JabRef details to your answer since a standalone JabRef answer would make a little contribution. I hope you don't mind. –  percusse Oct 13 '12 at 11:29
    
Surely. I've never used JabRef myself, but people praise it. –  Boris Oct 13 '12 at 18:20
    
JabRef offers a feature similar to bibexport: Tools > New Subdatabase based on AUX file. This, however, seems to work only with bibtex but not biblatex (I might be wrong here, it didn't work when I checked on my files, but I didn't check in detail why). –  Alex Dec 10 '12 at 9:56
    
It's worth noting that bibexport, and I think the other solutions that are being mentioned, won't work if you're using BibLaTeX because they rely on parsing .aux files. I've had a go at writing something that parses .bcf files, but it's very rough and ready: github.com/twsh/Bibliography-extraction. –  Tom Sep 17 '13 at 23:59

This is a python code that automatically deal with some of these issues. Please feel free to give feedback or suggestions to enhance or extend the features of this code. Hope it can help.

# perform some tests on ped.bib related to the pdf-directory (pdfdir)
# activate one or more tests by setting those control variables to 1

# 1. is_check_file: check objects with no or empty file entry
# 2. is_check_double_file: check if two or more different objects have the same pdf entry
# 3. is_check_unused_files: check <pdfdir> for unsused files

import os, sys, glob, copy
from os import path, access, R_OK  # W_OK for write permission.
from operator import itemgetter

#-------- control parameter ---------
is_check_file = 0
is_check_double_file = 0
is_check_unused_files = 1
#------------------------------------   

ROOT = os.getenv("HOME") # Home directory
#ROOT = path.expanduser("~") # works on all platforms  

pdfdir = ROOT + '/lit/pdf/'
print 'pdfdir', pdfdir
debug = 0 # shutdown debug/info messages

bib_data = open('ped.bib')

words = ['author', 'title', 'journal', 'year', 'volume', 'comment', 'issue', 'owner', 'file', 'timestamp', 'booktitle', 'editor', 'publisher', 'number', 'part', 'keywords', 'doi', 'month', 'organization', 'url']
#------------------------------------------------------------------------------

def check_missing_pdf(element):
    """
    following testes:
    1. element has no file entry
    2. element has empty file entry
    3. element has file entry with inexistant pdf-file
    """
    #check missing files
    if not element.has_key('file'): # element has no file
        print >>sys.stderr, '==== %s has no file entry'%element.get('key')
    elif not element.get('file'):  # .. or file is empty
            print >>sys.stderr, '**** %s has empty file entry'%element.get('key')
    else:
        # we have a file entry -----> check its existance in the pdf dir
        pdffile = pdfdir + element.get('file')
        if not( path.exists(pdffile) and path.isfile(pdffile) and access(pdffile, R_OK)):
            print >>sys.stderr, '#### %s with file entry [%s]: Either file is missing or is not readable'%(element.get('key'), element.get('file'))

#------------------------------------------------------------------------------

def check_doubles(elements):
    """
    check if two or more different objects have the same pdf-file 
    """
    doubles = {}
    for element in elements:
        pdffile = element.get("file")
        key = element.get("key")
        if not doubles.has_key(pdffile):
            doubles[pdffile] = [key]
        else:
            doubles[pdffile].append(key)
    for f, k in doubles.iteritems():
        if f and len(k) > 1: # if f excludes case f == None
            print >>sys.stderr, "Keys:", k, "have the same file <%s>"%f
#------------------------------------------------------------------------------

def check_unused_files(elements):
    """
    check dir pdfdir ("lit/pdf") for pdf-files that are not used
    in the ped.bib
    """
    pdf_files = glob.glob( pdfdir + "*.pdf") # pdfs in lit/pdf/
    dummy_files = copy.copy(pdf_files) # list of unused files
    for pdf in pdf_files:
        for element in elements:
            element_pdf =  element.get("file")
            if element_pdf is None:
                continue
            element_pdf = path.basename(element_pdf)

            if path.basename(pdf) == element_pdf:
                dummy_files.remove(pdf)
                break # check next files

    if dummy_files:
        print >>sys.stderr, "%d files are not used:"%len(dummy_files)
        for f in dummy_files:
            print >>sys.stderr, "---->",path.basename(f)
#------------------------------------------------------------------------------

def putWord(string, dic, line):
    """
    extract from <line> the value of the key <string> and put it in  <dic>
    """
    tmp = line[1].strip(' { } , .').split(':')
    # some files are like this :llll:aaaa. So tmp[0] is here == ''
    if not tmp[0]: 
        dic[string] = tmp[1]
    else:
        dic[string] = tmp[0]
#------------------------------------------------------------------------------

def getElement(f):
    """
    get ONE element from file f.
    return dict
    """
    dic = {}
    for line in f:
        line = line.strip(' \n\r')
        if not line:
            continue
        #get <key> and <type>
        if line[0] == '@':
            sline = line.split('{')
            typ = sline[0][1:]
            if typ == 'comment': # ignore jabref-meta
                continue
            dic['type'] = typ.strip(',') 
            key = sline[1].strip(',')
            if debug:
                print >> sys.stderr, '--------> type: <%s>'%typ
                print >> sys.stderr, '--------> key: <%s>'%key
            dic['key'] = key

        line = line.split('=')

        for word in words:
            if line[0].strip(' ') == word:
                putWord(word, dic, line)
                if debug:
                    print  >> sys.stderr, '--------> %s: <%s>'%(word, line[1].strip(' { },.') ) 

        # check for last line of element
        if line[0] == '}':
            if debug:
                print >> sys.stderr, '---------------------------------'
            return dic
#------------------------------------------------------------------------------


#----------------------- get content of file in elements ------------------------------
elements = []
while True:
    dic = getElement(bib_data) 
    if not dic:
        sorted(elements, key=itemgetter('key')) 
        break
    elements.append( dic )
#------------------------------------------------------------------------------

if is_check_file:
    print "check missing files ..."
    for element in elements:
        check_missing_pdf(element)

if is_check_double_file:
    print "check double files ..."
    check_doubles(elements)

if is_check_unused_files:
    print "is_check_unused_files ..."
    check_unused_files(elements)
share|improve this answer

The python script BibTeX Check helps cleaning up a bib file, though doing it not automatically but only providing a list of problems. It points to duplicate entries, missing fields, and inconsistent entries. When extending the script, it may reorder the list of entries or filter only the necessary ones.

share|improve this answer

Totally hackable python script:

https://bitbucket.org/adam-hammouda/bibtools-py

Right now it only sorts items with respect to the bibtex entry key. My preffered format is First-Author-Lastname. So it would sort things w.r.t author last name, assuming only that your keys are setup as such.

share|improve this answer

I have checked out several other scripts that are designed to clean the bib file. In addition to other tools mentioned here, bibtexformat

Generates and adds the labels (cite keys) to a BibTeX library file, abbreviates journal titles, filters out unneeded items, performs string replacements and checks author names for format errors that may lead to incorrect citations.

And BibTool by Gerd Neugebauer can extract bibliography easily from .aux file.

share|improve this answer
    
I just tried this scrip and it works great! It cleaned a bib file that had gone from papers to sente to bookends and removed all the unwanted crap and formatted it just as I wanted it. The defaults are very good, and it only took a bit of editing the confit file to customize it the way I wanted it. –  David Oct 12 at 4:28

Again duplicates can be very easily handled using JabRef's (I am using 2.10) File --> Import into Current Database. It checks for duplicates and all duplicates can be deselected easily by using "Deselect all Duplicates" button.

JabRef Import window

All duplicate entries would be maked as "D". Also answered here

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.