33

Is there a tool that will read a .bib file and add the correct DOI fields for all the entries that don't have them?

My current workflow is to just add citations as and when I need them with auctex in emacs. So a command line tool would be fine. I'd rather not load the file into some bibliography manager like jabref, because it adds all these superfluous fields like "owner" and "timestamp" which are useless. I guess with all the bibliographic information in the file, it should be easy to identify the right DOI with some judicious database searching...

7
  • AFAIK, this is a much harder problem than it initially seems. I've seen several discussions concerning various aspects of the same issue and never have I seen anything that comes close to a solution. Dec 9, 2010 at 14:22
  • AFAIK, if you open & save a bibtex file in Jabref, it does not add the fields "owner" and "timestamp". It merely re-formats the bibliography file and re-orders the entries so that everything looks consistent, but it does not add new fields. The "owner" and "timestamp" fields are only added for entries that you create using Jabref. Dec 9, 2010 at 14:46
  • Those databases often requires having a subscription, no? Other than EndNote which is tied into ISI, I'm not sure what other bibliography mangers have access to the suitable databases? Dec 9, 2010 at 14:58
  • 1
    I stand corrected. Mendeley does have bulk DOI look-up feedback.mendeley.com/forums/4941-mendeley-feedback/suggestions/… if you use their web account. Dec 9, 2010 at 15:02
  • @Willie: this might be best as an answer, as it does seem to cover at least some of the matter in hand
    – Joseph Wright
    Dec 9, 2010 at 15:37

7 Answers 7

21

I followed user13348's suggestion, and using his request function, I wrote a python3 script that takes a bib file and outputs a new bibfile with the DOIs it finds. I'm not using bibtool or taking any aux files.

The requirements are bibtexparser and unidecode.

#!/usr/bin/env python
import sys, re
from unidecode import unidecode
import bibtexparser
from bibtexparser.bwriter import BibTexWriter
import http.client as httplib
import urllib

# Search for the DOI given a title; e.g.  "computation in Noisy Radio Networks"
# Credit to user13348, slight modifications
# http://tex.stackexchange.com/questions/6810/automatically-adding-doi-fields-to-a-hand-made-bibliography
def searchdoi(title, author):
  params = urllib.parse.urlencode({"titlesearch":"titlesearch", "auth2" : author, "atitle2" : title, "multi_hit" : "on", "article_title_search" : "Search", "queryType" : "author-title"})
  headers = {"User-Agent": "Mozilla/5.0" , "Accept": "text/html", "Content-Type" : "application/x-www-form-urlencoded", "Host" : "www.crossref.org"}
  # conn = httplib.HTTPConnection("www.crossref.org:80") # Not working any more, HTTPS required
  conn = httplib.HTTPSConnection("www.crossref.org")       
  conn.request("POST", "/guestquery/", params, headers)
  response = conn.getresponse()
  #print(response.status, response.reason)
  data = response.read()
  conn.close()
  return re.search(r'doi\.org/([^"^<^>]+)', str(data))

def normalize(string):
    """Normalize strings to ascii, without latex."""
    string = re.sub(r'[{}\\\'"^]',"", string)
    string = re.sub(r"\$.*?\$","",string) # better remove all math expressions
    return unidecode(string)

def get_authors(entry):
    """Get a list of authors' or editors' last names."""
    def get_last_name(authors):
        for author in authors :
            author = author.strip(" ")
            if "," in author:
                yield author.split(",")[0]
            elif " " in author:
                yield author.split(" ")[-1]
            else:
                yield author

    try:
        authors = entry["author"]
    except KeyError:
        authors = entry["editor"]

    authors = normalize(authors).split("and")
    return list(get_last_name(authors))


print("Reading Bibliography...")
with open(sys.argv[1]) as bibtex_file:
    bibliography = bibtexparser.load(bibtex_file)


print("Looking for Dois...")
before = 0
new = 0
total = len(bibliography.entries)
for i,entry in enumerate(bibliography.entries):
    print("\r{i}/{total} entries processed, please wait...".format(i=i,total=total),flush=True,end="")
    try:
        if "doi" not in entry or entry["doi"].isspace():
            title = entry["title"]
            authors = get_authors(entry)
            for author in authors:
                doi_match = searchdoi(title,author)
                if doi_match:
                    doi = doi_match.groups()[0]
                    entry["doi"] = doi
                    new += 1
        else:
            before += 1
    except:
        pass
print("")

template="We added {new} DOIs !\nBefore: {before}/{total} entries had DOI\nNow: {after}/{total} entries have DOI"

print(template.format(new=new,before=before,after=before+new,total=total))
outfile = sys.argv[1]+"_doi.bib"
print("Writing result to ",outfile)
writer = BibTexWriter()
writer.indent = '    '     # indent entries with 4 spaces instead of one
with open(outfile, 'w') as bibfile:
    bibfile.write(writer.write(bibliography))

You can use it as such :

python3 searchdoi.py test.bib

And it will look like this :

Reading Bibliography...
Looking for Dois...
161/162 entries processed, please wait...
We added 49 DOIs !
Before: 42/162 entries had DOI
Now: 91/162 entries have DOI
Writing result to  test.bib_doi.bib

You can now just check test.bib_doi.bib.

3
  • Works like Charm! Thanks :) Aug 13, 2019 at 1:02
  • This script was a life saver for me, thanks a lot!
    – baccandr
    Aug 12, 2021 at 18:27
  • A remark: for an idea of a more robust way to match bibliography entries to Crossref entries, one can look at the S2ORC ACL paper aclanthology.org/2020.acl-main.447.pdf section 2.6
    – a3nm
    Jan 18, 2022 at 19:27
14

I wrote a short program that takes the BIB+AUX files and outputs an HTML file with all DOIs, obtained by querying CrossRef; see the code below. It would be great to modify the program so that the DOIs are written back to the original BIB file. If anyone is willing to help with this, please let me know!

The program uses bibtool, as well as the Python package zs.bibtex.parser. Once these are installed, use the following files (ignore bibtool's warnings on recursion limit):

1) "finddoi.sh"

bibtool -x $1.aux -o temp.bib -r formatting.txt
python finddoi.py temp.bib | tee doi_output.html

2) "formatting.txt" (used by bibtool)

key.number.separator = ""
fmt.et.al = ""
key.format = {short}
expand.macros = ON
delete.field {month}
print.line.length = 1000
print.braces = OFF
fmt.name.name = ""
new.format.type = { 17="%l " }
rewrite.rule { author # ".*" # "\"%100.17p(author)\"" }

3) "finddoi.py"

#!/usr/bin/env python

import httplib, urllib, re, sys, cgi
from zs.bibtex.parser import parse_string

# Search for the DOI given a title; e.g.  "computation in Noisy Radio Networks"
def searchdoi(title, author):
  params = urllib.urlencode({"titlesearch":"titlesearch", "auth2" : author, "atitle2" : title, "multi_hit" : "on", "article_title_search" : "Search", "queryType" : "author-title"})
  headers = {"User-Agent": "Mozilla/5.0" , "Accept": "text/html", "Content-Type" : "application/x-www-form-urlencoded", "Host" : "www.crossref.org"}
  conn = httplib.HTTPConnection("www.crossref.org:80")
  conn.request("POST", "/guestquery/", params, headers)
  response = conn.getresponse()
  # print response.status, response.reason
  data = response.read()
  conn.close()
  return data


# Main body

f = open(sys.argv[1], 'r')

inputdata = f.read()

# remove any leftover commas otherwise Bibtex parser crashed
inputdata = re.sub(r",(\s*})",r"\1", inputdata)

try:
  bibliography = parse_string(inputdata)
except:
  err = sys.exc_info()[1]
  print "Unexpected parsing error:", err
  sys.exit()

for paper in bibliography:
  try:
    title = bibliography[paper]['title']
    author = bibliography[paper]['author']
    if (isinstance(author,list)):
      author = author[0]
    author = str(author)
    author = re.sub(r"[{}'\\]","", author)
    # remove any of the characters that might confuse CrossRef
    title = re.sub(r"[{}]","", title)
    title = re.sub(r"\$.*?\$","",title) # better remove all math expressions
    title = re.sub(r"[^a-zA-Z0-9 ]", " ", title)
    print "<h1>DOIs for:<br>Title: %s<br>Author: %s<br> </h1>" % (title, author)
    out = searchdoi(title,author)
    result = re.findall(r"\<table cellspacing=1 cellpadding=1 width=600 border=0\>.*?\<\/table\>" ,out, re.DOTALL)
    if (len(result) > 0):
      print(result[0])
    else:
      print("Bad response from server<br><br>")
  except:
    print "Error with: ", bibliography[paper]
6

Building on the work of XaraB, here is a working implementation that uses the crossref API to get JSON objects directly. It also has some basic error handling and some retry logic to work around a current bug in the crossref API which causes queries to sometimes fail.

#!/usr/bin/env python
import sys
import re
from unidecode import unidecode
import bibtexparser
from bibtexparser.bwriter import BibTexWriter
import requests
import urllib
import time

# Search for the DOI given a title; e.g.  "computation in Noisy Radio Networks"
# Credit to user13348, slight modifications
# http://tex.stackexchange.com/questions/6810/automatically-adding-doi-fields-to-a-hand-made-bibliography
#


class DOIError(Exception):
    pass


def searchdoi(title, author, tries=4):
    params = urllib.parse.urlencode(
        {"query.author": author, "query.title": title})
    url_base = "http://api.crossref.org/works?"
    trying = True
    try_count = 0
    while trying and try_count <= tries:
        response = requests.get(url_base + params)
        if response.ok:
            trying = False
            try:
                doi = response.json()['message']['items'][0]['DOI']
            except:
                print("something wrong with json response for " + params)
                raise DOIError
        else:
            try_count += 1
            print("Response not 200 OK. Retrying, try " + str(try_count)
                + " of " + str(tries))
            time.sleep(1)
    if try_count >= tries:
        raise DOIError("Tried more than " + str(tries) + " times. Response"
                    " still not 200 OK! Uh oh...")
    return doi
#print(response.status, response.reason)


def normalize(string):
    """Normalize strings to ascii, without latex."""
    string = re.sub(r'[{}\\\'"^]', "", string)
    # better remove all math expressions
    string = re.sub(r"\$.*?\$", "", string)
    return unidecode(string)


def get_authors(entry):
    """Get a list of authors' or editors' last names."""
    def get_last_name(authors):
        for author in authors:
            author = author.strip(" ")
            if "," in author:
                yield author.split(",")[0]
            elif " " in author:
                yield author.split(" ")[-1]
            else:
                yield author

    try:
        authors = entry["author"]
    except KeyError:
        authors = entry["editor"]

    authors = normalize(authors).split("and")
    return list(get_last_name(authors))


def main(bibtex_filename):
    print("Reading Bibliography...")
    with open(bibtex_filename) as bibtex_file:
        bibliography = bibtexparser.load(bibtex_file)

    print("Looking for Dois...")
    before = 0
    new = 0
    total = len(bibliography.entries)
    for i, entry in enumerate(bibliography.entries):
        print("\r{i}/{total} entries processed, please wait...".format(i=i,
                                                                    total=total), flush=True, end="")
        try:
            if "doi" not in entry or entry["doi"].isspace():
                title = entry["title"]
                authors = entry["author"]
                try:
                    doi = searchdoi(title, authors)
                    entry["doi"] = doi
                    new += 1
                except DOIError:
                    print("unable to find DOI for " + title)
            else:
                before += 1
        except KeyError:
            print("some issue with this entry! No title or no author")
    print("")

    template = "We added {new} DOIs !\nBefore: {before}/{total} entries had DOI\nNow: {after}/{total} entries have DOI"

    print(
        template.format(
            new=new,
            before=before,
            after=before+new,
        total=total))
    outfile = bibtex_filename + "_doi.bib"
    print("Writing result to ", outfile)
    writer = BibTexWriter()
    writer.indent = '    '     # indent entries with 4 spaces instead of one
    with open(outfile, 'w') as bibfile:
        bibfile.write(writer.write(bibliography))

if __name__ == '__main__':
    main(sys.argv
4

Building on the work of XaraB, I changed the Httplib library to requests library which solve the redirect problem I was facing with Httplib. Previosly, the script looks for DOI by trying every author name even if the DOI is already found. So in the following script I skip to next bib entry if the DOI is already found which makes the script a bit faster.

#!/usr/bin/env python
import sys, re
from unidecode import unidecode
import bibtexparser
from bibtexparser.bwriter import BibTexWriter
import http.client as httplib
import requests
import urllib

# Search for the DOI given a title; e.g.  "computation in Noisy Radio Networks"
# Credit to user13348, slight modifications
# http://tex.stackexchange.com/questions/6810/automatically-adding-doi-fields-to-a-hand-made-bibliography
def searchdoi_using_requests(title, author):
    print("Searching for",title, author)
    params = {"auth2" : author, "atitle2" : title, "multi_hit" : "on", "article_title_search" : "Search", "queryType" : "author-title"}
    headers = {"User-Agent": "Mozilla/5.0" , "Accept": "text/html", "Content-Type" : "application/x-www-form-urlencoded", "Host" : "www.crossref.org"}
    url = "https://www.crossref.org/guestquery/#bibsearch"

    r = requests.post(url, headers=headers, data=params)

    data = r.text

    return re.search(r'doi\.org/([^"^<^>]+)', str(data))


def normalize(string):
    """Normalize strings to ascii, without latex."""
    string = re.sub(r'[{}\\\'"^]',"", string)
    string = re.sub(r"\$.*?\$","",string) # better remove all math expressions
    return unidecode(string)

def get_authors(entry):
    """Get a list of authors' or editors' last names."""
    def get_last_name(authors):
        for author in authors :
            author = author.strip(" ")
            if "," in author:
                yield author.split(",")[0]
            elif " " in author:
                yield author.split(" ")[-1]
            else:
                yield author

    try:
        authors = entry["author"]
    except KeyError:
        authors = entry["editor"]

    authors = normalize(authors).split("and")
    return list(get_last_name(authors))


print("Reading Bibliography...")
with open(sys.argv[1]) as bibtex_file:
    bibliography = bibtexparser.load(bibtex_file)


print("Looking for Dois...")
before = 0
new = 0
total = len(bibliography.entries)
for i,entry in enumerate(bibliography.entries):
    print("\r{i}/{total} entries processed, please wait...".format(i=i,total=total),flush=True,end="")
    try:
        if "doi" not in entry or entry["doi"].isspace():
            title = normalize(entry["title"])
            authors = get_authors(entry)
            for author in authors:
                doi_match = searchdoi_using_requests(title,author)
                if doi_match:
                    doi = doi_match.groups()[0]
                    entry["doi"] = doi
                    new += 1
                    break
        else:
            before += 1
    except:
        pass
print("")

template="We added {new} DOIs !\nBefore: {before}/{total} entries had DOI\nNow: {after}/{total} entries have DOI"

print(template.format(new=new,before=before,after=before+new,total=total))
outfile = sys.argv[1]+"_doi.bib"
print("Writing result to ",outfile)
writer = BibTexWriter()
writer.indent = '    '     # indent entries with 4 spaces instead of one
with open(outfile, 'w') as bibfile:
    bibfile.write(writer.write(bibliography))
2
  • Excellent, that saved me so much work! One remark: The script fails to find a doi if the title field itself contains braces (as in title = {Self-adjoint grommets in {B}anach spaces},); it works after removing the braces by hand, though. (A similar thing happens with inline math in titles, but that might be difficult to look up anyway.) Feb 14, 2018 at 11:59
  • 1
    Thanks for the comment! I have modified the code to remove special characters like braces from the titles. I have not tested the code myself but it should work.
    – arman
    Apr 10, 2018 at 14:22
2

I would use Mendeley desktop, import bib, add DOI, export bib. Bib output is very clean!

5
  • I can't find how to add DOI. The "search" button on the DOI field doesn't seem to work, and I can't find a bulk add DOI option.
    – Seamus
    Dec 9, 2010 at 16:45
  • 1
    It seems the doi lookup function is not working atm (grayed out for me). When i add PDFs, normally it looks up all information including the DOI. I just did a test and exported my Medeley library to which i manually added some DOIs to some of the articles, the DOI tag appears just fine in the resulting bib file. Now I wonder about the DOI lookup function of Medeley.. i'm almost sure it worked for me earlier.
    – Timtico
    Dec 9, 2010 at 16:59
  • It seems DOI's are only extracted from a few article types when using the PDF import function. for example BMC Bioinformatics articles have a DOI in their headings and this is picked up by Mendeley after importing, however many other articles dont have this. If you're ok with abit of programming you could build a an XML query interface and use 'crossref.org' to obtain the DOI's... perhaps somebody already build such an application!
    – Timtico
    Dec 9, 2010 at 17:08
  • crossref.org/guestquery is the link to where you can place your DOI XML queries
    – Timtico
    Dec 9, 2010 at 17:11
  • I like that idea! I don't have the knowledge or the time to do it, but having a little program that extracts information from the bib file and queries crossref.org seems like it should really already exist!
    – Seamus
    Dec 15, 2010 at 18:04
2

I wrote a python package to autocomplete bibtex entries (including but no limited to DOIs) inspired by thando's answer. It improves upon it by using the API instead of just searching the page, and querying multiple databases (crossref, DBLP, researchr and unpaywall)

You can install it with pip install bibtexautocomplete and run it with btac my_bib.bib.

If you only want to complete DOI's and not the other fields use the btac -c doi my_bib.bib.

You can find out more on the package's github or PyPi pages.

1

For completeness, let me point out that EPTCS has a service to do this: http://eptcs.web.cse.unsw.edu.au/doi/

Click on "Where can I find the DOIs of my references?".

The system allows you to submit a BibTeX file, and tries to search for the references it contains (apparently by searching in Crossref), and gives you a DOI for each reference (or tells you it did not find one). However, the service does not prepare a modified BibTeX file: you need to add back the references manually.

I do not think that this is better than the top answers, but just in case -- I also do not see the source code so I do not know exactly how the search and matching works, and if it is more or less clever than the code of the top answers.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .