TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there a tool that will read a .bib file and add the correct DOI fields for all the entries that don't have them?

My current workflow is to just add citations as and when I need them with auctex in emacs. So a command line tool would be fine. I'd rather not load the file into some bibliography manager like jabref, because it adds all these superfluous fields like "owner" and "timestamp" which are useless. I guess with all the bibliographic information in the file, it should be easy to identify the right DOI with some judicious database searching...

share|improve this question
AFAIK, this is a much harder problem than it initially seems. I've seen several discussions concerning various aspects of the same issue and never have I seen anything that comes close to a solution. – Yossi Farjoun Dec 9 '10 at 14:22
AFAIK, if you open & save a bibtex file in Jabref, it does not add the fields "owner" and "timestamp". It merely re-formats the bibliography file and re-orders the entries so that everything looks consistent, but it does not add new fields. The "owner" and "timestamp" fields are only added for entries that you create using Jabref. – Jukka Suomela Dec 9 '10 at 14:46
Those databases often requires having a subscription, no? Other than EndNote which is tied into ISI, I'm not sure what other bibliography mangers have access to the suitable databases? – Willie Wong Dec 9 '10 at 14:58
I stand corrected. Mendeley does have bulk DOI look-up feedback.mendeley.com/forums/4941-mendeley-feedback/suggestions/… if you use their web account. – Willie Wong Dec 9 '10 at 15:02
@Willie: this might be best as an answer, as it does seem to cover at least some of the matter in hand – Joseph Wright Dec 9 '10 at 15:37

I wrote a short program that takes the BIB+AUX files and outputs an HTML file with all DOIs, obtained by querying CrossRef; see the code below. It would be great to modify the program so that the DOIs are written back to the original BIB file. If anyone is willing to help with this, please let me know!

The program uses bibtool, as well as the Python package zs.bibtex.parser. Once these are installed, use the following files (ignore bibtool's warnings on recursion limit):

1) "finddoi.sh"

bibtool -x $1.aux -o temp.bib -r formatting.txt
python finddoi.py temp.bib | tee doi_output.html

2) "formatting.txt" (used by bibtool)

key.number.separator = ""
fmt.et.al = ""
key.format = {short}
expand.macros = ON
delete.field {month}
print.line.length = 1000
print.braces = OFF
fmt.name.name = ""
new.format.type = { 17="%l " }
rewrite.rule { author # ".*" # "\"%100.17p(author)\"" }

3) "finddoi.py"

#!/usr/bin/env python

import httplib, urllib, re, sys, cgi
from zs.bibtex.parser import parse_string

# Search for the DOI given a title; e.g.  "computation in Noisy Radio Networks"
def searchdoi(title, author):
  params = urllib.urlencode({"titlesearch":"titlesearch", "auth2" : author, "atitle2" : title, "multi_hit" : "on", "article_title_search" : "Search", "queryType" : "author-title"})
  headers = {"User-Agent": "Mozilla/5.0" , "Accept": "text/html", "Content-Type" : "application/x-www-form-urlencoded", "Host" : "www.crossref.org"}
  conn = httplib.HTTPConnection("www.crossref.org:80")
  conn.request("POST", "/guestquery/", params, headers)
  response = conn.getresponse()
  # print response.status, response.reason
  data = response.read()
  return data

# Main body

f = open(sys.argv[1], 'r')

inputdata = f.read()

# remove any leftover commas otherwise Bibtex parser crashed
inputdata = re.sub(r",(\s*})",r"\1", inputdata)

  bibliography = parse_string(inputdata)
  err = sys.exc_info()[1]
  print "Unexpected parsing error:", err

for paper in bibliography:
    title = bibliography[paper]['title']
    author = bibliography[paper]['author']
    if (isinstance(author,list)):
      author = author[0]
    author = str(author)
    author = re.sub(r"[{}'\\]","", author)
    # remove any of the characters that might confuse CrossRef
    title = re.sub(r"[{}]","", title)
    title = re.sub(r"\$.*?\$","",title) # better remove all math expressions
    title = re.sub(r"[^a-zA-Z0-9 ]", " ", title)
    print "<h1>DOIs for:<br>Title: %s<br>Author: %s<br> </h1>" % (title, author)
    out = searchdoi(title,author)
    result = re.findall(r"\<table cellspacing=1 cellpadding=1 width=600 border=0\>.*?\<\/table\>" ,out, re.DOTALL)
    if (len(result) > 0):
      print("Bad response from server<br><br>")
    print "Error with: ", bibliography[paper]
share|improve this answer

I would use Mendeley desktop, import bib, add DOI, export bib. Bib output is very clean!

share|improve this answer
I can't find how to add DOI. The "search" button on the DOI field doesn't seem to work, and I can't find a bulk add DOI option. – Seamus Dec 9 '10 at 16:45
It seems the doi lookup function is not working atm (grayed out for me). When i add PDFs, normally it looks up all information including the DOI. I just did a test and exported my Medeley library to which i manually added some DOIs to some of the articles, the DOI tag appears just fine in the resulting bib file. Now I wonder about the DOI lookup function of Medeley.. i'm almost sure it worked for me earlier. – Timtico Dec 9 '10 at 16:59
It seems DOI's are only extracted from a few article types when using the PDF import function. for example BMC Bioinformatics articles have a DOI in their headings and this is picked up by Mendeley after importing, however many other articles dont have this. If you're ok with abit of programming you could build a an XML query interface and use 'crossref.org' to obtain the DOI's... perhaps somebody already build such an application! – Timtico Dec 9 '10 at 17:08
crossref.org/guestquery is the link to where you can place your DOI XML queries – Timtico Dec 9 '10 at 17:11
I like that idea! I don't have the knowledge or the time to do it, but having a little program that extracts information from the bib file and queries crossref.org seems like it should really already exist! – Seamus Dec 15 '10 at 18:04

I followed user13348's suggestion, and using his request function, I wrote a python3 script that takes a bib file and outputs a new bibfile with the DOIs it finds. I'm not using bibtool or taking any aux files.

The requirements are bibtexparser and unidecode.

#!/usr/bin/env python
import sys, re
from unidecode import unidecode
import bibtexparser
from bibtexparser.bwriter import BibTexWriter
import http.client as httplib
import urllib

# Search for the DOI given a title; e.g.  "computation in Noisy Radio Networks"
# Credit to user13348, slight modifications
# http://tex.stackexchange.com/questions/6810/automatically-adding-doi-fields-to-a-hand-made-bibliography
def searchdoi(title, author):
  params = urllib.parse.urlencode({"titlesearch":"titlesearch", "auth2" : author, "atitle2" : title, "multi_hit" : "on", "article_title_search" : "Search", "queryType" : "author-title"})
  headers = {"User-Agent": "Mozilla/5.0" , "Accept": "text/html", "Content-Type" : "application/x-www-form-urlencoded", "Host" : "www.crossref.org"}
  conn = httplib.HTTPConnection("www.crossref.org:80")
  conn.request("POST", "/guestquery/", params, headers)
  response = conn.getresponse()
  #print(response.status, response.reason)
  data = response.read()
  return re.search(r'doi\.org/([^"^<^>]+)', str(data))

def normalize(string):
    """Normalize strings to ascii, without latex."""
    string = re.sub(r'[{}\\\'"^]',"", string)
    string = re.sub(r"\$.*?\$","",string) # better remove all math expressions
    return unidecode(string)

def get_authors(entry):
    """Get a list of authors' or editors' last names."""
    def get_last_name(authors):
        for author in authors :
            author = author.strip(" ")
            if "," in author:
                yield author.split(",")[0]
            elif " " in author:
                yield author.split(" ")[-1]
                yield author

        authors = entry["author"]
    except KeyError:
        authors = entry["editor"]

    authors = normalize(authors).split("and")
    return list(get_last_name(authors))

print("Reading Bibliography...")
with open(sys.argv[1]) as bibtex_file:
    bibliography = bibtexparser.load(bibtex_file)

print("Looking for Dois...")
before = 0
new = 0
total = len(bibliography.entries)
for i,entry in enumerate(bibliography.entries):
    print("\r{i}/{total} entries processed, please wait...".format(i=i,total=total),flush=True,end="")
        if "doi" not in entry or entry["doi"].isspace():
            title = entry["title"]
            authors = get_authors(entry)
            for author in authors:
                doi_match = searchdoi(title,author)
                if doi_match:
                    doi = doi_match.groups()[0]
                    entry["doi"] = doi
                    new += 1
            before += 1

template="We added {new} DOIs !\nBefore: {before}/{total} entries had DOI\nNow: {after}/{total} entries have DOI"

outfile = sys.argv[1]+"_doi.bib"
print("Writing result to ",outfile)
writer = BibTexWriter()
writer.indent = '    '     # indent entries with 4 spaces instead of one
with open(outfile, 'w') as bibfile:

You can use it as such :

python3 searchdoi.py test.bib

And it will look like this :

Reading Bibliography...
Looking for Dois...
161/162 entries processed, please wait...
We added 49 DOIs !
Before: 42/162 entries had DOI
Now: 91/162 entries have DOI
Writing result to  test.bib_doi.bib

You can now just check test.bib_doi.bib.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.