21

Is there a tool that will read a .bib file and add the correct DOI fields for all the entries that don't have them?

My current workflow is to just add citations as and when I need them with auctex in emacs. So a command line tool would be fine. I'd rather not load the file into some bibliography manager like jabref, because it adds all these superfluous fields like "owner" and "timestamp" which are useless. I guess with all the bibliographic information in the file, it should be easy to identify the right DOI with some judicious database searching...

  • AFAIK, this is a much harder problem than it initially seems. I've seen several discussions concerning various aspects of the same issue and never have I seen anything that comes close to a solution. – Yossi Farjoun Dec 9 '10 at 14:22
  • AFAIK, if you open & save a bibtex file in Jabref, it does not add the fields "owner" and "timestamp". It merely re-formats the bibliography file and re-orders the entries so that everything looks consistent, but it does not add new fields. The "owner" and "timestamp" fields are only added for entries that you create using Jabref. – Jukka Suomela Dec 9 '10 at 14:46
  • Those databases often requires having a subscription, no? Other than EndNote which is tied into ISI, I'm not sure what other bibliography mangers have access to the suitable databases? – Willie Wong Dec 9 '10 at 14:58
  • 1
    I stand corrected. Mendeley does have bulk DOI look-up feedback.mendeley.com/forums/4941-mendeley-feedback/suggestions/… if you use their web account. – Willie Wong Dec 9 '10 at 15:02
  • @Willie: this might be best as an answer, as it does seem to cover at least some of the matter in hand – Joseph Wright Dec 9 '10 at 15:37
12

I wrote a short program that takes the BIB+AUX files and outputs an HTML file with all DOIs, obtained by querying CrossRef; see the code below. It would be great to modify the program so that the DOIs are written back to the original BIB file. If anyone is willing to help with this, please let me know!

The program uses bibtool, as well as the Python package zs.bibtex.parser. Once these are installed, use the following files (ignore bibtool's warnings on recursion limit):

1) "finddoi.sh"

bibtool -x $1.aux -o temp.bib -r formatting.txt
python finddoi.py temp.bib | tee doi_output.html

2) "formatting.txt" (used by bibtool)

key.number.separator = ""
fmt.et.al = ""
key.format = {short}
expand.macros = ON
delete.field {month}
print.line.length = 1000
print.braces = OFF
fmt.name.name = ""
new.format.type = { 17="%l " }
rewrite.rule { author # ".*" # "\"%100.17p(author)\"" }

3) "finddoi.py"

#!/usr/bin/env python

import httplib, urllib, re, sys, cgi
from zs.bibtex.parser import parse_string

# Search for the DOI given a title; e.g.  "computation in Noisy Radio Networks"
def searchdoi(title, author):
  params = urllib.urlencode({"titlesearch":"titlesearch", "auth2" : author, "atitle2" : title, "multi_hit" : "on", "article_title_search" : "Search", "queryType" : "author-title"})
  headers = {"User-Agent": "Mozilla/5.0" , "Accept": "text/html", "Content-Type" : "application/x-www-form-urlencoded", "Host" : "www.crossref.org"}
  conn = httplib.HTTPConnection("www.crossref.org:80")
  conn.request("POST", "/guestquery/", params, headers)
  response = conn.getresponse()
  # print response.status, response.reason
  data = response.read()
  conn.close()
  return data


# Main body

f = open(sys.argv[1], 'r')

inputdata = f.read()

# remove any leftover commas otherwise Bibtex parser crashed
inputdata = re.sub(r",(\s*})",r"\1", inputdata)

try:
  bibliography = parse_string(inputdata)
except:
  err = sys.exc_info()[1]
  print "Unexpected parsing error:", err
  sys.exit()

for paper in bibliography:
  try:
    title = bibliography[paper]['title']
    author = bibliography[paper]['author']
    if (isinstance(author,list)):
      author = author[0]
    author = str(author)
    author = re.sub(r"[{}'\\]","", author)
    # remove any of the characters that might confuse CrossRef
    title = re.sub(r"[{}]","", title)
    title = re.sub(r"\$.*?\$","",title) # better remove all math expressions
    title = re.sub(r"[^a-zA-Z0-9 ]", " ", title)
    print "<h1>DOIs for:<br>Title: %s<br>Author: %s<br> </h1>" % (title, author)
    out = searchdoi(title,author)
    result = re.findall(r"\<table cellspacing=1 cellpadding=1 width=600 border=0\>.*?\<\/table\>" ,out, re.DOTALL)
    if (len(result) > 0):
      print(result[0])
    else:
      print("Bad response from server<br><br>")
  except:
    print "Error with: ", bibliography[paper]
8

I followed user13348's suggestion, and using his request function, I wrote a python3 script that takes a bib file and outputs a new bibfile with the DOIs it finds. I'm not using bibtool or taking any aux files.

The requirements are bibtexparser and unidecode.

#!/usr/bin/env python
import sys, re
from unidecode import unidecode
import bibtexparser
from bibtexparser.bwriter import BibTexWriter
import http.client as httplib
import urllib

# Search for the DOI given a title; e.g.  "computation in Noisy Radio Networks"
# Credit to user13348, slight modifications
# http://tex.stackexchange.com/questions/6810/automatically-adding-doi-fields-to-a-hand-made-bibliography
def searchdoi(title, author):
  params = urllib.parse.urlencode({"titlesearch":"titlesearch", "auth2" : author, "atitle2" : title, "multi_hit" : "on", "article_title_search" : "Search", "queryType" : "author-title"})
  headers = {"User-Agent": "Mozilla/5.0" , "Accept": "text/html", "Content-Type" : "application/x-www-form-urlencoded", "Host" : "www.crossref.org"}
  # conn = httplib.HTTPConnection("www.crossref.org:80") # Not working any more, HTTPS required
  conn = httplib.HTTPSConnection("www.crossref.org")       
  conn.request("POST", "/guestquery/", params, headers)
  response = conn.getresponse()
  #print(response.status, response.reason)
  data = response.read()
  conn.close()
  return re.search(r'doi\.org/([^"^<^>]+)', str(data))

def normalize(string):
    """Normalize strings to ascii, without latex."""
    string = re.sub(r'[{}\\\'"^]',"", string)
    string = re.sub(r"\$.*?\$","",string) # better remove all math expressions
    return unidecode(string)

def get_authors(entry):
    """Get a list of authors' or editors' last names."""
    def get_last_name(authors):
        for author in authors :
            author = author.strip(" ")
            if "," in author:
                yield author.split(",")[0]
            elif " " in author:
                yield author.split(" ")[-1]
            else:
                yield author

    try:
        authors = entry["author"]
    except KeyError:
        authors = entry["editor"]

    authors = normalize(authors).split("and")
    return list(get_last_name(authors))


print("Reading Bibliography...")
with open(sys.argv[1]) as bibtex_file:
    bibliography = bibtexparser.load(bibtex_file)


print("Looking for Dois...")
before = 0
new = 0
total = len(bibliography.entries)
for i,entry in enumerate(bibliography.entries):
    print("\r{i}/{total} entries processed, please wait...".format(i=i,total=total),flush=True,end="")
    try:
        if "doi" not in entry or entry["doi"].isspace():
            title = entry["title"]
            authors = get_authors(entry)
            for author in authors:
                doi_match = searchdoi(title,author)
                if doi_match:
                    doi = doi_match.groups()[0]
                    entry["doi"] = doi
                    new += 1
        else:
            before += 1
    except:
        pass
print("")

template="We added {new} DOIs !\nBefore: {before}/{total} entries had DOI\nNow: {after}/{total} entries have DOI"

print(template.format(new=new,before=before,after=before+new,total=total))
outfile = sys.argv[1]+"_doi.bib"
print("Writing result to ",outfile)
writer = BibTexWriter()
writer.indent = '    '     # indent entries with 4 spaces instead of one
with open(outfile, 'w') as bibfile:
    bibfile.write(writer.write(bibliography))

You can use it as such :

python3 searchdoi.py test.bib

And it will look like this :

Reading Bibliography...
Looking for Dois...
161/162 entries processed, please wait...
We added 49 DOIs !
Before: 42/162 entries had DOI
Now: 91/162 entries have DOI
Writing result to  test.bib_doi.bib

You can now just check test.bib_doi.bib.

6

Building on the work of XaraB, here is a working implementation that uses the crossref API to get JSON objects directly. It also has some basic error handling and some retry logic to work around a current bug in the crossref API which causes queries to sometimes fail.

#!/usr/bin/env python
import sys
import re
from unidecode import unidecode
import bibtexparser
from bibtexparser.bwriter import BibTexWriter
import requests
import urllib
import time

# Search for the DOI given a title; e.g.  "computation in Noisy Radio Networks"
# Credit to user13348, slight modifications
# http://tex.stackexchange.com/questions/6810/automatically-adding-doi-fields-to-a-hand-made-bibliography
#


class DOIError(Exception):
    pass


def searchdoi(title, author, tries=4):
    params = urllib.parse.urlencode(
        {"query.author": author, "query.title": title})
    url_base = "http://api.crossref.org/works?"
    trying = True
    try_count = 0
    while trying and try_count <= tries:
        response = requests.get(url_base + params)
        if response.ok:
            trying = False
            try:
                doi = response.json()['message']['items'][0]['DOI']
            except:
                print("something wrong with json response for " + params)
                raise DOIError
        else:
            try_count += 1
            print("Response not 200 OK. Retrying, try " + str(try_count)
                + " of " + str(tries))
            time.sleep(1)
    if try_count >= tries:
        raise DOIError("Tried more than " + str(tries) + " times. Response"
                    " still not 200 OK! Uh oh...")
    return doi
#print(response.status, response.reason)


def normalize(string):
    """Normalize strings to ascii, without latex."""
    string = re.sub(r'[{}\\\'"^]', "", string)
    # better remove all math expressions
    string = re.sub(r"\$.*?\$", "", string)
    return unidecode(string)


def get_authors(entry):
    """Get a list of authors' or editors' last names."""
    def get_last_name(authors):
        for author in authors:
            author = author.strip(" ")
            if "," in author:
                yield author.split(",")[0]
            elif " " in author:
                yield author.split(" ")[-1]
            else:
                yield author

    try:
        authors = entry["author"]
    except KeyError:
        authors = entry["editor"]

    authors = normalize(authors).split("and")
    return list(get_last_name(authors))


def main(bibtex_filename):
    print("Reading Bibliography...")
    with open(bibtex_filename) as bibtex_file:
        bibliography = bibtexparser.load(bibtex_file)

    print("Looking for Dois...")
    before = 0
    new = 0
    total = len(bibliography.entries)
    for i, entry in enumerate(bibliography.entries):
        print("\r{i}/{total} entries processed, please wait...".format(i=i,
                                                                    total=total), flush=True, end="")
        try:
            if "doi" not in entry or entry["doi"].isspace():
                title = entry["title"]
                authors = entry["author"]
                try:
                    doi = searchdoi(title, authors)
                    entry["doi"] = doi
                    new += 1
                except DOIError:
                    print("unable to find DOI for " + title)
            else:
                before += 1
        except KeyError:
            print("some issue with this entry! No title or no author")
    print("")

    template = "We added {new} DOIs !\nBefore: {before}/{total} entries had DOI\nNow: {after}/{total} entries have DOI"

    print(
        template.format(
            new=new,
            before=before,
            after=before+new,
        total=total))
    outfile = bibtex_filename + "_doi.bib"
    print("Writing result to ", outfile)
    writer = BibTexWriter()
    writer.indent = '    '     # indent entries with 4 spaces instead of one
    with open(outfile, 'w') as bibfile:
        bibfile.write(writer.write(bibliography))

if __name__ == '__main__':
    main(sys.argv
4

Building on the work of XaraB, I changed the Httplib library to requests library which solve the redirect problem I was facing with Httplib. Previosly, the script looks for DOI by trying every author name even if the DOI is already found. So in the following script I skip to next bib entry if the DOI is already found which makes the script a bit faster.

#!/usr/bin/env python
import sys, re
from unidecode import unidecode
import bibtexparser
from bibtexparser.bwriter import BibTexWriter
import http.client as httplib
import requests
import urllib

# Search for the DOI given a title; e.g.  "computation in Noisy Radio Networks"
# Credit to user13348, slight modifications
# http://tex.stackexchange.com/questions/6810/automatically-adding-doi-fields-to-a-hand-made-bibliography
def searchdoi_using_requests(title, author):
    print("Searching for",title, author)
    params = {"auth2" : author, "atitle2" : title, "multi_hit" : "on", "article_title_search" : "Search", "queryType" : "author-title"}
    headers = {"User-Agent": "Mozilla/5.0" , "Accept": "text/html", "Content-Type" : "application/x-www-form-urlencoded", "Host" : "www.crossref.org"}
    url = "https://www.crossref.org/guestquery/#bibsearch"

    r = requests.post(url, headers=headers, data=params)

    data = r.text

    return re.search(r'doi\.org/([^"^<^>]+)', str(data))


def normalize(string):
    """Normalize strings to ascii, without latex."""
    string = re.sub(r'[{}\\\'"^]',"", string)
    string = re.sub(r"\$.*?\$","",string) # better remove all math expressions
    return unidecode(string)

def get_authors(entry):
    """Get a list of authors' or editors' last names."""
    def get_last_name(authors):
        for author in authors :
            author = author.strip(" ")
            if "," in author:
                yield author.split(",")[0]
            elif " " in author:
                yield author.split(" ")[-1]
            else:
                yield author

    try:
        authors = entry["author"]
    except KeyError:
        authors = entry["editor"]

    authors = normalize(authors).split("and")
    return list(get_last_name(authors))


print("Reading Bibliography...")
with open(sys.argv[1]) as bibtex_file:
    bibliography = bibtexparser.load(bibtex_file)


print("Looking for Dois...")
before = 0
new = 0
total = len(bibliography.entries)
for i,entry in enumerate(bibliography.entries):
    print("\r{i}/{total} entries processed, please wait...".format(i=i,total=total),flush=True,end="")
    try:
        if "doi" not in entry or entry["doi"].isspace():
            title = normalize(entry["title"])
            authors = get_authors(entry)
            for author in authors:
                doi_match = searchdoi_using_requests(title,author)
                if doi_match:
                    doi = doi_match.groups()[0]
                    entry["doi"] = doi
                    new += 1
                    break
        else:
            before += 1
    except:
        pass
print("")

template="We added {new} DOIs !\nBefore: {before}/{total} entries had DOI\nNow: {after}/{total} entries have DOI"

print(template.format(new=new,before=before,after=before+new,total=total))
outfile = sys.argv[1]+"_doi.bib"
print("Writing result to ",outfile)
writer = BibTexWriter()
writer.indent = '    '     # indent entries with 4 spaces instead of one
with open(outfile, 'w') as bibfile:
    bibfile.write(writer.write(bibliography))
  • Excellent, that saved me so much work! One remark: The script fails to find a doi if the title field itself contains braces (as in title = {Self-adjoint grommets in {B}anach spaces},); it works after removing the braces by hand, though. (A similar thing happens with inline math in titles, but that might be difficult to look up anyway.) – Christian Clason Feb 14 '18 at 11:59
  • 1
    Thanks for the comment! I have modified the code to remove special characters like braces from the titles. I have not tested the code myself but it should work. – JohnM Apr 10 '18 at 14:22
2

I would use Mendeley desktop, import bib, add DOI, export bib. Bib output is very clean!

  • I can't find how to add DOI. The "search" button on the DOI field doesn't seem to work, and I can't find a bulk add DOI option. – Seamus Dec 9 '10 at 16:45
  • 1
    It seems the doi lookup function is not working atm (grayed out for me). When i add PDFs, normally it looks up all information including the DOI. I just did a test and exported my Medeley library to which i manually added some DOIs to some of the articles, the DOI tag appears just fine in the resulting bib file. Now I wonder about the DOI lookup function of Medeley.. i'm almost sure it worked for me earlier. – Timtico Dec 9 '10 at 16:59
  • It seems DOI's are only extracted from a few article types when using the PDF import function. for example BMC Bioinformatics articles have a DOI in their headings and this is picked up by Mendeley after importing, however many other articles dont have this. If you're ok with abit of programming you could build a an XML query interface and use 'crossref.org' to obtain the DOI's... perhaps somebody already build such an application! – Timtico Dec 9 '10 at 17:08
  • crossref.org/guestquery is the link to where you can place your DOI XML queries – Timtico Dec 9 '10 at 17:11
  • I like that idea! I don't have the knowledge or the time to do it, but having a little program that extracts information from the bib file and queries crossref.org seems like it should really already exist! – Seamus Dec 15 '10 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.