Prevent `biber --tool` from removing non-standard fields in .bib files

Question

This question was about a side effect encountered in this answer. One way to avoid this side effect is to not use biber but bibtool -biblatex as stated in this answer.

I'm using biber --tool to automatically remove fields like abstract, review, groups, and file from biblatex entries in .bib files. A side effect of this is that all non-standard-fields¹ seem to be removed as well, which I want to prevent. Here's an minimal example:

This is the entry in the .bib file mybib.bib:

@Thesis{Author_18_TheThesis,
 author           = {Mr Author},
 title            = {The Thesis},
 type             = {Doctoral Dissertation},
 institution      = {Department of Documents, University of Stackexchange},
 year             = {2018},
 abstract         = {This is the abstract.},
 file             = {:author/Author_18_TheThesis.pdf:PDF},
 review           = {This is the review.},
 groups           = {publications},
 ispreprintpublic = {true},
}

And this is the clean-bibfiles.conf configartion file for biber --tool:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <output_fieldcase>lower</output_fieldcase>
    <output_indent>2</output_indent>
    <output_align>true</output_align>
    <sourcemap>
        <maps datatype="bibtex" map_overwrite="1">
            <map map_overwrite="1">
                <map_step map_field_set="abstract" map_null="1"/>
                <map_step map_field_set="review" map_null="1"/>
                <map_step map_field_set="groups" map_null="1"/>
                <map_step map_field_set="file" map_null="1"/>
            </map>
        </maps>
    </sourcemap>
</config>

Running biber --tool --configfile=clean-bibfiles.conf mybib.bib will produce a mybib_bibertool.bib file that contains this biblatex entry:

@thesis{Author_18_TheThesis,
  author      = {Author, Mr},
  institution = {Department of Documents, University of Stackexchange},
  date        = {2018},
  title       = {The Thesis},
  type        = {Doctoral Dissertation},
}

The non-standard-field ispreprintpublic has been stripped from the entry. To be precise, the groups and review fields would also be stripped even if they would not be in the sourcemap, as both are also non-standard fields or are "borrowed" from the article type, respectively. How to prevent the automatic stripping of such fields when using biber --tool?

---

¹See e.g. "2.1.1 Regular Types" and "4.2.4.1 Generic Fields" in the biblatex documentation.

Answer 1

Indeed, biber does not handle fields which are not part of the datamodel, so if your data source has non-standard fields or entrytypes, you have to provide biber with a data model which includes them.

But, before proceeding with this task, it is a good idea to know what is actually missing in the default datamodel that you need. You can do that with the --validate-datamodel option:

biber --tool --validate-datamodel mybib.bib

On a .bib file containing that entry of yours, that gives you the following warnings:

WARN - Datamodel: Entry 'Author_18_TheThesis' (mybib.bib): Field 'review' invalid in data model - ignoring
WARN - Datamodel: Entry 'Author_18_TheThesis' (mybib.bib): Field 'groups' invalid in data model - ignoring
WARN - Datamodel: Entry 'Author_18_TheThesis' (mybib.bib): Field 'ispreprintpublic' invalid in data model - ignoring

Which lets you know precisely that you have to include the fields review, groups and ispreprintpublic to the datamodel.

(Of course, adding review and groups to the datamodel to then drop them in the sourcemap would not be the "usual". I'm just doing it here for the sake of generality and procedure.)

To set your custom biber-tool.conf file (you called it clean-bibfiles.conf, let's keep it that way, but in general there are other operations which could be achieved through this procedure) you will have to adjust the default datamodel, which is in the default biber-tool.conf, which in turn can be found with:

biber --tool-config

That should return the location of the default biber-tool.conf. If you open that file, you will find the default data model specifications (everything between <datamodel> and </datamodel>). Copy that (yes, all that) in your clean-bibfiles.conf, just below your sourcemap specifications, as you had them originally defined. Then add the line(s) of your non-standard field(s) within the <fields>...</fields> group. In your case (assuming here these are "literal" type fields):

<field fieldtype="field" datatype="literal">review</field>
<field fieldtype="field" datatype="literal">groups</field>
<field fieldtype="field" datatype="literal">ispreprintpublic</field>

And, within the group <entryfields><entrytype>thesis</entrytype>...</entryfields> add:

<field>review</field>
<field>groups</field>
<field>ispreprintpublic</field>

The resulting clean-bibfiles.conf should have then the following general structure (I can't include it in its entirety, for it exceeds the limits of the site):

<?xml version="1.0" encoding="UTF-8"?>
<config>
  <output_fieldcase>title</output_fieldcase>
  <output_indent>2</output_indent>
  <output_align>true</output_align>
  <sourcemap>
    <maps datatype="bibtex" map_overwrite="1">
      <map map_overwrite="1">
        <map_step map_field_set="abstract" map_null="1"/>
        <map_step map_field_set="review" map_null="1"/>
        <map_step map_field_set="groups" map_null="1"/>
        <map_step map_field_set="file" map_null="1"/>
      </map>
    </maps>
  </sourcemap>
  <datamodel>
    <constants>
      ...
    </constants>
    <entrytypes>
      ...
    </entrytypes>
    <fields>
      <field fieldtype="field" datatype="literal">review</field>
      <field fieldtype="field" datatype="literal">groups</field>
      <field fieldtype="field" datatype="literal">ispreprintpublic</field>
      ...
    </fields>
    <entryfields>
      ...
    </entryfields>
    ...
    <entryfields>
      <entrytype>thesis</entrytype>
      <field>review</field>
      <field>groups</field>
      <field>ispreprintpublic</field>
      ...
    </entryfields>
    ...
    <constraints>
      ...
    </constraints>
  </datamodel>
</config>

With this configuration file, the command:

biber --tool --validate-datamodel --configfile=clean-bibfiles.conf mybib.bib

will output mybib_bibertool.bib with the entry as desired:

@Thesis{Author_18_TheThesis,
  Author           = {Author, Mr},
  Institution      = {Department of Documents, University of Stackexchange},
  Date             = {2018},
  Ispreprintpublic = {test},
  Title            = {The Thesis},
  Type             = {Doctoral Dissertation},
}

This may not be specially straightforward. But, to quote PLK's comment above: "The benefits of having a datamodel in tool mode outweigh this sort of problem."