Take the 2-minute tour ×
TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. It's 100% free, no registration required.

Is there a tool that will read a .bib file and add the correct DOI fields for all the entries that don't have them?

My current workflow is to just add citations as and when I need them with auctex in emacs. So a command line tool would be fine. I'd rather not load the file into some bibliography manager like jabref, because it adds all these superfluous fields like "owner" and "timestamp" which are useless. I guess with all the bibliographic information in the file, it should be easy to identify the right DOI with some judicious database searching...

share|improve this question
    
AFAIK, this is a much harder problem than it initially seems. I've seen several discussions concerning various aspects of the same issue and never have I seen anything that comes close to a solution. –  Yossi Farjoun Dec 9 '10 at 14:22
    
AFAIK, if you open & save a bibtex file in Jabref, it does not add the fields "owner" and "timestamp". It merely re-formats the bibliography file and re-orders the entries so that everything looks consistent, but it does not add new fields. The "owner" and "timestamp" fields are only added for entries that you create using Jabref. –  Jukka Suomela Dec 9 '10 at 14:46
    
Those databases often requires having a subscription, no? Other than EndNote which is tied into ISI, I'm not sure what other bibliography mangers have access to the suitable databases? –  Willie Wong Dec 9 '10 at 14:58
    
I stand corrected. Mendeley does have bulk DOI look-up feedback.mendeley.com/forums/4941-mendeley-feedback/suggestions/… if you use their web account. –  Willie Wong Dec 9 '10 at 15:02
    
@Willie: this might be best as an answer, as it does seem to cover at least some of the matter in hand –  Joseph Wright Dec 9 '10 at 15:37

2 Answers 2

I wrote a short program that takes the BIB+AUX files and outputs an HTML file with all DOIs, obtained by querying CrossRef; see the code below. It would be great to modify the program so that the DOIs are written back to the original BIB file. If anyone is willing to help with this, please let me know!

The program uses bibtool, as well as the Python package zs.bibtex.parser. Once these are installed, use the following files (ignore bibtool's warnings on recursion limit):

1) "finddoi.sh"

bibtool -x $1.aux -o temp.bib -r formatting.txt
python finddoi.py temp.bib | tee doi_output.html

2) "formatting.txt" (used by bibtool)

key.number.separator = ""
fmt.et.al = ""
key.format = {short}
expand.macros = ON
delete.field {month}
print.line.length = 1000
print.braces = OFF
fmt.name.name = ""
new.format.type = { 17="%l " }
rewrite.rule { author # ".*" # "\"%100.17p(author)\"" }

3) "finddoi.py"

#!/usr/bin/env python

import httplib, urllib, re, sys, cgi
from zs.bibtex.parser import parse_string

# Search for the DOI given a title; e.g.  "computation in Noisy Radio Networks"
def searchdoi(title, author):
  params = urllib.urlencode({"titlesearch":"titlesearch", "auth2" : author, "atitle2" : title, "multi_hit" : "on", "article_title_search" : "Search", "queryType" : "author-title"})
  headers = {"User-Agent": "Mozilla/5.0" , "Accept": "text/html", "Content-Type" : "application/x-www-form-urlencoded", "Host" : "www.crossref.org"}
  conn = httplib.HTTPConnection("www.crossref.org:80")
  conn.request("POST", "/guestquery/", params, headers)
  response = conn.getresponse()
  # print response.status, response.reason
  data = response.read()
  conn.close()
  return data


# Main body

f = open(sys.argv[1], 'r')

inputdata = f.read()

# remove any leftover commas otherwise Bibtex parser crashed
inputdata = re.sub(r",(\s*})",r"\1", inputdata)

try:
  bibliography = parse_string(inputdata)
except:
  err = sys.exc_info()[1]
  print "Unexpected parsing error:", err
  sys.exit()

for paper in bibliography:
  try:
    title = bibliography[paper]['title']
    author = bibliography[paper]['author']
    if (isinstance(author,list)):
      author = author[0]
    author = str(author)
    author = re.sub(r"[{}'\\]","", author)
    # remove any of the characters that might confuse CrossRef
    title = re.sub(r"[{}]","", title)
    title = re.sub(r"\$.*?\$","",title) # better remove all math expressions
    title = re.sub(r"[^a-zA-Z0-9 ]", " ", title)
    print "<h1>DOIs for:<br>Title: %s<br>Author: %s<br> </h1>" % (title, author)
    out = searchdoi(title,author)
    result = re.findall(r"\<table cellspacing=1 cellpadding=1 width=600 border=0\>.*?\<\/table\>" ,out, re.DOTALL)
    if (len(result) > 0):
      print(result[0])
    else:
      print("Bad response from server<br><br>")
  except:
    print "Error with: ", bibliography[paper]
share|improve this answer

I would use Mendeley desktop, import bib, add DOI, export bib. Bib output is very clean!

share|improve this answer
    
I can't find how to add DOI. The "search" button on the DOI field doesn't seem to work, and I can't find a bulk add DOI option. –  Seamus Dec 9 '10 at 16:45
1  
It seems the doi lookup function is not working atm (grayed out for me). When i add PDFs, normally it looks up all information including the DOI. I just did a test and exported my Medeley library to which i manually added some DOIs to some of the articles, the DOI tag appears just fine in the resulting bib file. Now I wonder about the DOI lookup function of Medeley.. i'm almost sure it worked for me earlier. –  Timtico Dec 9 '10 at 16:59
    
It seems DOI's are only extracted from a few article types when using the PDF import function. for example BMC Bioinformatics articles have a DOI in their headings and this is picked up by Mendeley after importing, however many other articles dont have this. If you're ok with abit of programming you could build a an XML query interface and use 'crossref.org' to obtain the DOI's... perhaps somebody already build such an application! –  Timtico Dec 9 '10 at 17:08
    
crossref.org/guestquery is the link to where you can place your DOI XML queries –  Timtico Dec 9 '10 at 17:11
    
I like that idea! I don't have the knowledge or the time to do it, but having a little program that extracts information from the bib file and queries crossref.org seems like it should really already exist! –  Seamus Dec 15 '10 at 18:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.