3

This question was about a side effect encountered in this answer. One way to avoid this side effect is to not use biber but bibtool -biblatex as stated in this answer.

I'm using biber --tool to automatically remove fields like abstract, review, groups, and file from biblatex entries in .bib files. A side effect of this is that all non-standard-fields¹ seem to be removed as well, which I want to prevent. Here's an minimal example:

This is the entry in the .bib file mybib.bib:

@Thesis{Author_18_TheThesis,
 author           = {Mr Author},
 title            = {The Thesis},
 type             = {Doctoral Dissertation},
 institution      = {Department of Documents, University of Stackexchange},
 year             = {2018},
 abstract         = {This is the abstract.},
 file             = {:author/Author_18_TheThesis.pdf:PDF},
 review           = {This is the review.},
 groups           = {publications},
 ispreprintpublic = {true},
}

And this is the clean-bibfiles.conf configartion file for biber --tool:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <output_fieldcase>lower</output_fieldcase>
    <output_indent>2</output_indent>
    <output_align>true</output_align>
    <sourcemap>
        <maps datatype="bibtex" map_overwrite="1">
            <map map_overwrite="1">
                <map_step map_field_set="abstract" map_null="1"/>
                <map_step map_field_set="review" map_null="1"/>
                <map_step map_field_set="groups" map_null="1"/>
                <map_step map_field_set="file" map_null="1"/>
            </map>
        </maps>
    </sourcemap>
</config>

Running biber --tool --configfile=clean-bibfiles.conf mybib.bib will produce a mybib_bibertool.bib file that contains this biblatex entry:

@thesis{Author_18_TheThesis,
  author      = {Author, Mr},
  institution = {Department of Documents, University of Stackexchange},
  date        = {2018},
  title       = {The Thesis},
  type        = {Doctoral Dissertation},
}

The non-standard-field ispreprintpublic has been stripped from the entry. To be precise, the groups and review fields would also be stripped even if they would not be in the sourcemap, as both are also non-standard fields or are "borrowed" from the article type, respectively. How to prevent the automatic stripping of such fields when using biber --tool?


¹See e.g. "2.1.1 Regular Types" and "4.2.4.1 Generic Fields" in the biblatex documentation.

  • 3
    I don't think this is possible at the moment without adding these fields to the data model. All unknown fields are dropped. If you need a different behaviour, start a feature request: github.com/plk/biber/issues – moewe Feb 12 '18 at 21:49
  • 2
    My solution would be to try BibTool, but of course your other question shows you tried it and that is why you ended up with Biber in the first place. I don't use BibTool, but the manual shows a possibility to define new entry types. And BibTool even has a biblatex mode that already knows @thesis. Run BibTool with bibtool -r biblatex <file.bib> – moewe Feb 12 '18 at 22:03
  • 2
    Ha found, a bit of the docs that describe this. Its in the changes section at the beginning and not in the section describing the tool mode: "Fields which are not defined in the data model described in the default biber-tool.conf are ignored and are neither read nor output. If custom fields are required, they should be defined in the data model by using a custom tool mode config file." This confirms my suspicion that it is indeed not possible without a data model definition. – moewe Feb 12 '18 at 22:15
  • 2
    1) Copy the default XML configuration file (to find it: biber --tool-config. 2) define a new field in the <fields> element: <field fieldtype="field" datatype="literal">ispreprintpublic</field>. 3) add it to the thesis entryfields: <field>ispreprintpublic</field>. 4) use your new configuration file... – Paul Gaborit Feb 13 '18 at 13:56
  • 3
    @PaulGaborit solution is correct.The benefits of having a datamodel in tool mode outweigh this sort of problem. You should add unknown fields to the datamodel by copying the default tool-mode config file. Also, you can force biber to report on skipping things unknown to the data model by using the --validate-datamodel option. – PLK Feb 13 '18 at 20:47
4

Indeed, biber does not handle fields which are not part of the datamodel, so if your data source has non-standard fields or entrytypes, you have to provide biber with a data model which includes them.

But, before proceeding with this task, it is a good idea to know what is actually missing in the default datamodel that you need. You can do that with the --validate-datamodel option:

biber --tool --validate-datamodel mybib.bib

On a .bib file containing that entry of yours, that gives you the following warnings:

WARN - Datamodel: Entry 'Author_18_TheThesis' (mybib.bib): Field 'review' invalid in data model - ignoring
WARN - Datamodel: Entry 'Author_18_TheThesis' (mybib.bib): Field 'groups' invalid in data model - ignoring
WARN - Datamodel: Entry 'Author_18_TheThesis' (mybib.bib): Field 'ispreprintpublic' invalid in data model - ignoring

Which lets you know precisely that you have to include the fields review, groups and ispreprintpublic to the datamodel.

(Of course, adding review and groups to the datamodel to then drop them in the sourcemap would not be the "usual". I'm just doing it here for the sake of generality and procedure.)

To set your custom biber-tool.conf file (you called it clean-bibfiles.conf, let's keep it that way, but in general there are other operations which could be achieved through this procedure) you will have to adjust the default datamodel, which is in the default biber-tool.conf, which in turn can be found with:

biber --tool-config

That should return the location of the default biber-tool.conf. If you open that file, you will find the default data model specifications (everything between <datamodel> and </datamodel>). Copy that (yes, all that) in your clean-bibfiles.conf, just below your sourcemap specifications, as you had them originally defined. Then add the line(s) of your non-standard field(s) within the <fields>...</fields> group. In your case (assuming here these are "literal" type fields):

<field fieldtype="field" datatype="literal">review</field>
<field fieldtype="field" datatype="literal">groups</field>
<field fieldtype="field" datatype="literal">ispreprintpublic</field>

And, within the group <entryfields><entrytype>thesis</entrytype>...</entryfields> add:

<field>review</field>
<field>groups</field>
<field>ispreprintpublic</field>

The resulting clean-bibfiles.conf should have then the following general structure (I can't include it in its entirety, for it exceeds the limits of the site):

<?xml version="1.0" encoding="UTF-8"?>
<config>
  <output_fieldcase>title</output_fieldcase>
  <output_indent>2</output_indent>
  <output_align>true</output_align>
  <sourcemap>
    <maps datatype="bibtex" map_overwrite="1">
      <map map_overwrite="1">
        <map_step map_field_set="abstract" map_null="1"/>
        <map_step map_field_set="review" map_null="1"/>
        <map_step map_field_set="groups" map_null="1"/>
        <map_step map_field_set="file" map_null="1"/>
      </map>
    </maps>
  </sourcemap>
  <datamodel>
    <constants>
      ...
    </constants>
    <entrytypes>
      ...
    </entrytypes>
    <fields>
      <field fieldtype="field" datatype="literal">review</field>
      <field fieldtype="field" datatype="literal">groups</field>
      <field fieldtype="field" datatype="literal">ispreprintpublic</field>
      ...
    </fields>
    <entryfields>
      ...
    </entryfields>
    ...
    <entryfields>
      <entrytype>thesis</entrytype>
      <field>review</field>
      <field>groups</field>
      <field>ispreprintpublic</field>
      ...
    </entryfields>
    ...
    <constraints>
      ...
    </constraints>
  </datamodel>
</config>

With this configuration file, the command:

biber --tool --validate-datamodel --configfile=clean-bibfiles.conf mybib.bib

will output mybib_bibertool.bib with the entry as desired:

@Thesis{Author_18_TheThesis,
  Author           = {Author, Mr},
  Institution      = {Department of Documents, University of Stackexchange},
  Date             = {2018},
  Ispreprintpublic = {test},
  Title            = {The Thesis},
  Type             = {Doctoral Dissertation},
}

This may not be specially straightforward. But, to quote PLK's comment above: "The benefits of having a datamodel in tool mode outweigh this sort of problem."

  • 1
    +1 for clarity and amount of detail. I think you meant to write </entryfields> instead of <\entryfields>. – geekoverdose Feb 16 '18 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.