4

My current workflow is

  1. edit CiteULike database on-line
  2. wget http://www.citeulike.org/bibtex/user/MYUSER/tag/MYTAG?key_type=4
  3. biber biblio
  4. pdflatex biblio

where biblio.tex is a file for a bibliography/reference list, generated with a \nocite{*}.

The problem is that the CiteULike algorithm for generating the AuthorYearTitle key (key_type=4 in the wget URL above) may generate collisions, i. e. two different entries with the same key.

While this is for sure a CiteULike bug, I would like to patch this problem by inserting another step between wget and biber that rewrites the .bib database resolving duplicate keys, inserting a trailing 'a', 'b', ... letter after the duplicated AuthorYearTitle key. (Edit: I would like to maintain an AuthorYearTitle key in case the bibliography needs further editing.)

I would like a solution where no user intervention is required, so that steps 2., 3., 4., ... can be run in a script.

I was not able to find an existing solution to this (simple?) problem, and before reinventing the wheel I would like to ask for advice on this forum.

Note

By omitting the key_type=4 parameters, one gets unique numeric keys (citeulike:123456), so a possible strategy is to use a more robust key generator and forget about the ones generated by CiteULike.

  • 1
    If you're willing to add another software into the loop Jabref has a duplicate resolving function – Elad Den May 10 '17 at 15:03
  • @EladDen is it possibile to use jabref duplicate resolution from the command line? – Stefano M May 10 '17 at 15:58
  • Not that I am aware of. Also, Jabref does ask you which of the two versions you wish to keep for each duplicate it finds (I know this might not be what you're looking for, therefore I did not put it in an answer) – Elad Den May 10 '17 at 16:51
4
+50

Since you are using \nocite{*}, you do not need the bib tags to represent anything meaningful. So an easy solution would very simply replace all tags in the .bib file by some random character sequence (or append one, see below). For me, this worked:

sed -i -e 's~^@\(.\+\){.\+,$~(echo @\1{; cat /dev/urandom | tr -dc "a-zA-Z0-9" | fold -w 8 | head -1; echo ,) | paste -sd ""~e' biblio.bib

What does this do?

sed 

well, call sed

-i 

do inline-replacement

-e 

use extended regular expressions (for capture-group and replacement)

'

begin sed command

s

substitute

~

substitution start delimiter

^@\(.\+\){.\+,$

Search for start of line, @, a group of non-whitespace followed by {, another group of non-whitespace, ,, end of line; e.g., @article{AuthorYear,

~

substitution middle delimiter

(echo @\1{; cat /dev/urandom | tr -dc "a-zA-Z0-9" | fold -w 8 | head -1; echo ,) | paste -sd ""

Replace by

  1. echo @, the first non-whitespace group captured earlier, {
  2. echo one random tag (cat through head)
  3. echo ,

(After removing newlines in between with paste)

~

substitution end delimiter

e

call the replacement as a shell command and use output as replacement text

'

end sed command

biblio.bib

filename

Update 1 If you want to keep the original tags, just capture them and append the random tag:

sed -i -e 's~^@\(.\+\){\(.\+\),$~(echo @\1{\2_; cat /dev/urandom | tr -dc "a-zA-Z0-9" | fold -w 8 | head -1; echo ,) | paste -sd ""~e' biblio.bib

Update 2 Instead of replacing by a random tag, this one add a, b, ... to the tags that were seen before; so the first occurrence of "AuthorYear" remains unchanged, the second becomes "AuthorYeara", and so forth.

echo > cache
sed -i -e 's~^@\(.\+\){\(.\+\),$~(echo @\1{\2; awk "BEGIN{printf \\\"%c\\\", 96+`grep \2 cache | wc -l`}" | sed "s/\\\`//"; echo \2 >> cache; echo ,) | paste -sd ""~e' biblio.bib

Can be changed to make the second "b", of course:

echo > cache
sed -i -e 's~^@\(.\+\){\(.\+\),$~(echo @\1{\2; awk "BEGIN{printf \\\"%c\\\", 97+`grep \2 cache | wc -l`}" | sed "s/a//"; echo \2 >> cache; echo ,) | paste -sd ""~e' biblio.bib
  • I would prefer to maintain the AuthorYearTitle keys, for later editing. Sorry for not having mentioned this in the question. – Stefano M May 5 '17 at 8:02
  • @StefanoM easy, see last update. – bers May 5 '17 at 8:08
  • I would like something more "stable": key disambiguation should be done only in case of collisions and two subsequent runs of the script should generate the same .bib database. – Stefano M May 5 '17 at 8:20
  • @StefanoM stability is easily achieved by piping /dev/urandom to a cache file once and using that cache file in subsequent runs instead of /dev/urandom; this way, two subsequence runs of the script will generate the same .bib file. On the other hand, I have no idea what "key disambiguation should be done only in case of collisions " has to do with "stability". – bers May 5 '17 at 8:56
  • @StefanoM "Update 2" solves "key disambiguation should be done only in case of collisions". – bers May 5 '17 at 9:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.