Extend data model to allow for custom fields with biber in tool mode

Question

I want to extend biber's data model, i.e. I want biber to accept fields such as review or owner as valid entry fields for all entry types.

^{(Note: This is not about using data from these fields in a document, nor do I want to map custom fields to existing fields such as usera or customa. Please see below ("Background") if you wonder why I'm asking this.)}

Using biber in tool mode, I extended the default configuration file (found with biber --tool-conf) in the following way:

<config>
<!-- skipped 383 lines -->
  <datamodel>
    <!-- skipped "constants" and "entrytypes" --> 
    <fields>
      <!-- skipped all predefined fields --> 
      <field fieldtype="field" datatype="literal">owner</field>
      <field fieldtype="field" datatype="literal">review</field>
    </fields>
    <entryfields>
      <field>owner</field>
      <field>review</field>
    <!-- skipped remainder of file -->
    </entryfields>
  </datamodel>
</config>

I uploaded the full modified biber.conf to pastebin.com.

As far as I can see, this "works", i.e. running biber --tool input.bib with this input.bib

@REPORT{Self2016,
  YEAR      = {2016},
  AUTHOR    = {Self, My},
  TITLE     = {The title},
  OWNER     = {Me},
  REVIEW    = {Important stuff.}
}

does not delete the review and owner fields with the modified configuration file (whereas the default configuration deletes these fields).

My Question is: Why does this work? I found this solution more or less by trial and error. I suppose that adding the <field ...> in <fields> tells biber that such a field may exist. Actually, this extension of the configuration alone is functional. I further assume that the entry under <entryfields> makes the new field a valid field for all entry types. If I skip this part, biber complains when I call it with --validate-datamodel.

Could you please tell me if my assumptions are correct and help me to understand where this can be found in the biber or biblatex manual? I think §4.5.4 in the biblatex manual is quite relevant, but is there any documentation on how the LaTeX commands documented there are related to the XML structure in biber.conf?

Besides, is this the correct way to extend the data model or did I do it terribly wrong but the test case is too simple to reveal this?

---

Background: I use JabRef to manage my references. Sometimes, I use JabRef's "Review" tab to annotate an entry. These notes are saved in the review field of the respective entry. From time to time, I run biber in tool mode on the BIB file, in order to get clean formatting, sorting and apply some other transformations. This worked well in the past, but as of biber 2.6, biber discards custom entry fields. This is what I reported here on Github where @plk advised me to extend the data model.

Answer 1

You are correct in your assumptions. The field elements describe allowable fields. Biber uses this to decide whether to allow the field. Other aspects of the datamodel are used for validation. The .conf file datamodel format is not really documented because people rarely use custom datamodels in tool mode. The biblatex macros for datamodel manipulation write the XML you see to the .bcf file (but with a bcf: namespace).

Answer 2

I had the exact same problem, and I really didn't want to maintain my own version of the Biber config file. I noticed that in the default data model there are the many fields defined which I never use, including usera, userb etc. and lista, listb etc. So my workaround is to rename my custom fields (in this case fields starting with OPT, which I want to preserve) to these pre-defined fields, run Biber in tool mode, then rename them back. Here's a little Bash script which does this:

# Define replacements (each will be prefixed with 'OPT')
FIND=(address booktitle doi   editor isbn  issn  location month note  url)
REPL=(usera   userb     userc userd  usere userf lista    listb listc listd)
IXS=`seq 0 $((${#FIND[*]} - 1))`

# Input and output file
FILE=refs.bib

# Replace forward
for i in $IXS; do
  sed -i '' s/OPT${FIND[$i]}/${REPL[$i]}/ $FILE
done

# Biber
biber --tool --output-file=$FILE $FILE

# Replace back
for i in $IXS; do
  sed -i '' s/${REPL[$i]}/OPT${FIND[$i]}/ $FILE
done