Command-line tools for some bibtex database manipulations

Question

Inspired by an answer to another question, I am reminded of the various reasons why it's difficult to have only one BIB file. I would like to have a set of command line tools that I could script to take my 'master' BIB and convert it for a specific situation. I am aware of bibtool which is probably powerful enough that it can meet most of my needs, but I haven't figured out how to use it...

Here are some common tasks that could be automated this way. Of course, the strategic goal would be solved 'the right way' by using a smarter BST (or something like biblatex) but often one is constrained (or at least it is much more convenient) to use a publisher's broken BST and make a few tweaks to the database:

1. Truncate all papers with more than 5 authors, and keep only the first 3 authors of any truncated papers (because you are running low on space)
2. Remove a field (month, url, ISSN, ISBN) because a journal has a broken BST that makes a mess of these
3. Remove a field conditionally (eg, remove title only for articles but not books)
4. Expand/collapse @string references (eg, I have 2 files, one apsjour.bib with
   @STRING{prl = {Phys. Rev. Lett.}}
   and one fulljour.bib with
   @STRING{prl = {Physical Review Letters}}
   so I can write \bibliography{apsjour,articlebib} or \bibliography{fulljour,articlebib} as required, but I certainly don't expect my co-authors to deal with this convention.
5. Remove archiveprefix, eprint, primaryclass from @articles with a page number
6. Remove DOIs with weird characters in them because the documentclass doesn't do enough escaping (seriously, Wiley, is
   10.1002/(SICI)1521-3978(200005)48:5/7<531::AID-PROP531>3.0.CO;2-#
   really a good idea for something that needs to end up in a URL?)
7. A tool that is flexible enough to do more sophisticated things would be nice, so long as the flexibility isn't at the price of usability (cf bibtool)
8. There are probably other use cases that I've temporarily forgotten. I feel like I'm forever making journal-specific or even paper-specific modifications of bibtex databases. JabRef makes it fairly easy, but still...

There are lots of cute tools living on CTAN in tex-archive/biblio/bibtex/utils but I feel that there must be some other place where the serious tools hide. I can't be the only one with these problems, can I? (Feel free to tell me that I'm approaching this the wrong way and let me know your personal strategy for dealing with the above issues without using commandline tools! This includes, as Jukka Suomela suggests in his answer, tools for editing the generated BBL file instead of editing the BIB.)

Here are some problems that I already know of solutions for:

1. Process an AUX file to keep only the cited entries (many solutions, including bibextract)
2. Merge BIB files and remove duplicates (pybib)
3. Force a particular format of page ranges (1234-56 or 1234-1256) (bibfile-reformat-pages)

Answer 1

I do not know any non-programmer tools that can do those things you ask for, so I think you will need a bib scripting toolkit, and pybib is the most commonly used of those. If you are not familiar with python but do know perl, then btOOL may be more suited to you.

Answer 2

I have never had a situation in which I must provide a Bibtex file that compiles flawlessly with a journal-supplied BST file. Do you really need to do that?

For conference and journal submissions, one typically just sends a PDF file, not the source code. For the final version, in my experience it's usually enough to provide a Latex source code in which I have already replaced the \bibliography command by the contents of the Bibtex-generated *.bbl file.

Nobody will see how I created the *.bbl file, so I can do anything I want:

• Take the journal-supplied BST file and hack it a bit. It may be much faster to simply remove some buggy parts from the BST file than to tweak your Bibtex database. This makes a lot of sense if you need to deal with the same buggy BST file frequently.

• Just take the final *.bbl file and edit it directly. Particularly useful to handle some one-off special cases.

Another trick that I have used to handle some very exceptional cases is to have two entries in my Bibtex database for the same paper (obviously with different keys, like "foo" and "foo2"). This is useful if you'd like to decide case-by-case which version to use in which paper, independently of other issues such as different BST files.