# Use \addbibresource via a script

Is there some way to use \addbibresource with biblatex to fetch a remote bibliography (in my case from CiteULike), but piping it through a script first?

The output from CiteULike isn't perfect for what I need, so at the moment I use a bash script to grab it, pipe it through sed to strip out junk and correct a few things, and then standard BibTeX to assemble the bibliography for me.

A minimum working example doesn't seem to make sense in this case, so I'll pin my script instead.

#!/bin/bash

curl "http://www.citeulike.org/bibtex/user/MarkEveritt/tag/nv_notes" | sed \
-e 's/pages = {$$.*$$-.*--.*-.*},/pages = {\1},/' \
-e 's/pages = {$$.*$$+},/pages = {\1},/' \
-e 's/comment =/note =/' \
-e 's/\\\$/$/g' \
-e 's/$$^.*title.*$$\\_/\1_/' \
-e '/^[ ]*abstract.*/d' \
-e '/^[ ]*citeulike-.*/d' \
-e '/^[ ]*priority.*/d' \
-e '/^[ ]*posted-at.*/d' \
-e '/^[ ]*keywords.*/d' \
> bibliography.bib

-
You can compile via pdflatex --shell-escpape and then you can use the script inside LaTeX. –  Marco Daniel Nov 9 '11 at 14:38
@MarcoDaniel: Hmm, it'd work, but seems crude. I may end up doing this anyway, but if I did I'd only want it to happen for when I run BibTeX as it takes a few seconds for curl to grab the .bib file. Any ideas? –  qubyte Nov 9 '11 at 14:43

You can ignore/rename/map fields using biber (see section 3.1.1 of the biber manual). This covers most of your use cases as it allows you to rename and delete fields. Changing the values of fields is now supported in a general way.

Here is how to, for example, remove "+" characters in the PAGES field:

<?xml version="1.0" encoding="UTF-8"?>
<config>
<sourcemap>
<maps datatype="bibtex" bmap_overwrite="1">
<map>
<map_step map_field_source="PAGES" map_match="\+" map_replace=""/>
</map>
</maps>
</sourcemap>
</config>


Edited to show the Biber 0.9.8 format for this option

-
As you've seen from my script, ignoring fields is half the battle. The rest is cleaning up CiteULike output, which has a nasty habit of adding slashes and odd characters when only the start page is known (which happens a lot as online publications frequently use just the start page). Are you the biber maintainer? I look forward to using it! –  qubyte Nov 10 '11 at 2:38
Yes, I'm the biber maintainer. Tomorrow I will upload a new beta of biber which will also allow full Perl regexp modification of fields so you should be able to cover all of your example. I'll update this thread when it's uploaded. It will be in biber 0.9.6/biblatex 1.7 which will be out soon. –  PLK Nov 10 '11 at 22:15
Wow. That's amazing! And it'll be an incentive for me to learn a bit of Perl too. Thanks! –  qubyte Nov 10 '11 at 22:21
You can try this out now with the 0.9.6 beta just uploaded to SF: sourceforge.net/projects/biblatex-biber/files/biblatex-biber/… you'll need to be using the biblatex 1.7 beta .sty from here: sourceforge.net/projects/biblatex-biber/files/biblatex.sty See section 3.1.1 of the biber PDF doc for instructions on setting up your biber.conf file to do this. The Perl regular expressions are not so different from the sed ones and you should be able to implement the example above quite easily. –  PLK Nov 11 '11 at 9:11
By the way, in case you didn't know, you can also directly access remote Citeulike bibs with biber. See the biblatex manual section 3.5.1 - you can do things like: \addbibresource[location=remote]{citeulike.org/bibtex/group/9517} –  PLK Nov 11 '11 at 17:54

First of all you cannot use \addbibresource with remote resource if you use biblatex with BibTeX. Remote resources are only supported by biber. biblatex itself does only add the link to the remote resource to the configuration file, that will be used from biber. biber does not support commands to download remote resources. And it does not support post download processing. So using biber instead of BibTeX would not help.

But what you can do is:

1. Make a new LaTeX command, that writes remote source information to the aux file.
2. Write a script, that reads the remote source information from the aux file, calls curl, pipes the output and runs BibTeX

Here's a complete example using BibTeX and simple unsrt style instead of biblatex:

\begin{filecontents*}{mybibtex.sh}
#!/bin/sh
#
# $1 is a aux file echo "$1"
for arb in "grep '\\remotebibresource {' "$1""; do echo "$arb"
remote=echo "$arb" | cut -b 21- | cut -d\} -f 1 echo "$remote"
lfile=echo "$arb" | cut -b 21- | cut -d\} -f 2 | cut -b 2- echo "$lfile"
curl "$remote" | sed \ -e 's/pages = {$$.*$$-.*--.*-.*},/pages = {\1},/' \ -e 's/pages = {$$.*$$+},/pages = {\1},/' \ -e 's/comment =/note =/' \ -e 's/\\\$/$/g' \ -e 's/$$^.*title.*$$\\_/\1_/' \ -e '/^[ ]*abstract.*/d' \ -e '/^[ ]*citeulike-.*/d' \ -e '/^[ ]*priority.*/d' \ -e '/^[ ]*posted-at.*/d' \ -e '/^[ ]*keywords.*/d' \ > "$lfile"
done

bibtex "\$1"
\end{filecontents*}
\documentclass{article}
\bibliographystyle{unsrt}

\makeatletter
\protected@write\@auxout{}{\protect\remotebibresource{#1}{#2}}%
\bibliography{#2}%
}
\newcommand*{\remotebibresource}[2]{}
\AtBeginDocument{%
\protected@write\@auxout{}{\global\let\protect\remotebibresource\protect\@gobbletwo}%
}
\makeatother

\begin{document}
\nocite{*}
Test
\end{document}


After the first run, you first have to make mybibtex.sh executable and then run mybibtex.sh <aux-file>.

After two more runs of latex you will get:

To make it work with biblatex and biber and \addbibresource you just have to replace \bibliography by \addbibresource and call of bibtex (at mybibtex.sh) by call of biber. And for sure you have to change the example document to use biblatex ;-).

The script itself my be improved. But it works and my already be usefull for some users. That's the reason, I've made this improved example.

Last but not least: There's a good support for biblatex and biber. Maybe a feature request to support user defined remote resource types (to run curl) and user defined commands to preprocess a (temporary) database before biber processes would be a good idea.

-
I'm aware that I can't use BibTeX with \addbibresource. At the moment I'm constrained to BibTeX because I have to use REVTeX, but I was considering using biber as a backend to biblatex for other stuff if it was capable of doing this. As it is, I'll use my own script for now and continue with BibTeX as there is not enough incentive to fragment my workflow when I have a set of scripts (i.e. one that calls the one above and BibTeX) that already does the job. –  qubyte Nov 9 '11 at 15:29
@Mark S. Everitt: Have a look at my changed example. It does not need biber and even works without biblatex but very simple BibTeX style. But the changes for using biblatex are very simple and suggestions for this are part of the answer. Maybe you could accept this as an answer to your question. –  Schweinebacke Nov 9 '11 at 16:15
Hmm, this is interesting - I might see about adding field changing via regexps to the biber <map></map> option. Since it's all perl internally, you would be able to use arbitrary perl regexps to manipulate fields. –  PLK Nov 9 '11 at 16:50
@Schweinebacke: Sorry, I'm in Tokyo so it was time to sleep. I'm going to hold off accepting an answer for now, as this answer is 95% what I already do. If nothing else comes up (regular expressions etc.) then I'll be happy to accept it. –  qubyte Nov 10 '11 at 0:00