3

I have for three decades now kept notes in my BibTeX file. Unfortunately, I have to use biblatex in order to get footcite. I just want biblatex to skip over anything, that is not a properly formatted BibTeX entry. If there's a filter, I can use to just extract all the proper BibTeX entries from my file, I don't mind taking the time to make an intermediate file.

Note that BibTeX, once it recognises it can't parse something, will bail and look for the next newline-@.

That's all I need and can't believe, it hasn't already been written.

4

There are several reasons why Biber is a bit more fussy about .bib files than BibTeX. I'll list a few possible stumbling blocks for Biber that are usually not that much of an issue (if at all) for BibTeX and where possible try to explain why there is a difference and try to suggest a cure.

Before we plunge in, we should note that people often use Biber synonymous with biblatex+Biber and BibTeX synonymous with \bibliographystyle+BibTeX (for the differences see also bibtex vs. biber and biblatex vs. natbib and What to do to switch to biblatex?). biblatex can be used with BibTeX as backend instead of Biber, even though that is explicitly discouraged now, since many advanced features are only available with Biber. Some of the limitations discussed here are limitations of biblatex and its approach to bibliographies, those will occur regardless of the backend (BibTeX or Biber). Some limitations are specifically linked to Biber (the program) and will not appear with BibTeX (even in a biblatex setting). In the following I will try to make clear if something is a biblatex issue or a Biber issue, but I will always refer to the alternative as BibTeX, even though there should probably be a distinction between BibTeX (the program) and 'the BibTeX way of producing bibliographies on a fundamental level/the LaTeX side'. There is quite some variability in the .bst styles available for BibTeX, in the following we will usually assume a style that works similar to plain.bst or one of the natbib standard styles.

The general relevance and job description of BibTeX and Biber are discussed brilliantly in Question mark or bold citation key instead of citation number, so I suggest to check out that post if you are not quite sure why you even need to run BibTeX or Biber. The short of it is that BibTeX and Biber make the data in the .bib file accessible to LaTeX and perform some processing steps that would be hard to do directly in LaTeX (with BibTeX that is mainly sorting and some label generation, Biber also calculates uniqueness information and a few other handy things). The LaTeX-digestible data is written to the .bbl file.

.bbl files

Where BibTeX (in the sense of the BibTeX way of producing bibliographies with .bst files/\bibliographystyle, as for example used with natbib) and biblatex differ fundamentally is in the format of the LaTeX-digestible .bbl file they produce. The .bbl file produced by BibTeX usually contains text that is ready-to-typeset in a thebibliography environment. If one were to write the bibliography manually, one would probably come up with code that looks similar to the output produced by BibTeX. (There are some BibTeX styles that go a bit further and use more complex macros to allow the user to change more aspects of the style on the fly. You can read more about this and see some examples in the background section of my answer to Customize citation with bibulous.) The .bbl file is simply read in and typeset as normal LaTeX code by the \bibliography command and produces a complete bibliography (LaTeX doesn't know about the contents of the entries, it just typesets them). biblatex's .bbl files, on the other hand, don't contain ready-to-typeset code, they contain the complete entry data and some additional info in a structured and LaTeX-readable way (again, an example of what that looks like can be found in Customize citation with bibulous). The .bbl for biblatex is read at the beginning of the document in its entirety so that all entry data is available throughout the entire document (so LaTeX 'knows' about the entry data of each item).

Invalid LaTeX code in fields

The difference between .bbl files already explains a first reason for why BibTeX is more tolerant about certain things than biblatex. With the BibTeX approach only the data that is actually supposed to printed in the bibliography is passed on to LaTeX (modulo jurabib), while with biblatex all known fields are passed on to LaTeX. Issues associated with malformed abstract fields often fall into the category of problems caused by this fact. Sometimes the abstract field of .bib files exported from a website contains invalid LaTeX code (cf. Software-generated bibliographic entries: common errors and other mistakes to check before use), for example an unescaped % or # character. For BibTeX users this is usually not an issue, since the abstract field is rarely typeset in the bibliography and thus rarely included in the .bbl file and seen by LaTeX. BibTeX unlike LaTeX does not have any issues with unescaped % and #, so everything goes smoothly. But for biblatex abstract is a known field, which means that it is passed on to LaTeX and read in at the beginning of the document. Unescaped # and % can the wreak havoc and cause errors. The field is read in even if it is never used and even if the user does not intend to use it. A possible remedy here is to tell Biber to remove the abstract field before it gets to the .bbl file, see Error with percent sign in bib entry field when using biblatex/biber.

btparse

Biber is based on the Perl module Text::BibTeX, which in turn relies on the C library btparse to parse .bib files. There are some subtle differences between what btparse accepts as valid input and what BibTeX accepts. For example btparse does not allow () in entry keys, but BibTeX is fine with them (see Using BibTeX keys containing parentheses with Biber).

BibTeX sometimes still tries to produce output even when there is an error whereas Biber will simply abort processing the entry in question or worse the complete run. Often the BibTeX output will be nonsensical in those cases, but the BibTeX error message is not very intrusive and I can imagine it may fall through the cracks especially when your editor digests the log messages for you. There is not a lot you can do here save for rewriting Biber or the underlying parsing library.

File encodings

BibTeX only understands ASCII, whereas Biber has full Unicode support and can read files in other encodings as well. When BibTeX encounters non-ASCII chars, they are usually passed through unchanged, but in some situations (sorting mainly) the result may not be as desired. Biber on the other hand, can read and understand files in a variety of encodings (non-ASCII chars can be sorted according to a specified locale etc.). That means that the .bib files must be correctly encoded in the encoding expected by Biber (usually biblatex assumes the .bib file has the same encoding as the .tex file as declared to inputenc or dictated by the Engine; of course the automatic detection can be overridden with bibencoding): Files with mixed encoding may fail or come out as garbage. In this case the only solution is to make sure the file uses the specified encoding correctly.

--nodieonerror

Biber has the option --nodieonerror which allows it to proceed past some errors that would normally abort a Biber run. In practice this is helpful for a certain class of errors produced by the underlying btparse library, which sometimes can be skipped over, but it may well happen that going past an error causes a crash or nonsensical output. So this is something I would use only for debugging and I wouldn't expect wonders.

  • Thanks! This is great! My problem is indeed that I have accreted multiple file encodings over time. Is there a tool that will quickly strip out anything that isn't just ascii, like bibtex does? Any citation I actually use I have converted to ascii mark up, I never rely on unicode. – Joanna Bryson Jul 24 '19 at 14:56
  • @JoannaBryson BibTeX doesn't strip anything non-ASCII it just passes it through unchanged. That means that sorting based on those characters is probably wrong and means that things will explode if LaTeX can't deal with those characters. Since Biber can't handle malformed files you need to remove the incorrect encoding with an external tool before Biber processes the file. I assume there are many ways to do that (I would have probably used Notepad++), e.g. unix.stackexchange.com/q/475548. Just search the web a bit. – moewe Jul 24 '19 at 15:43
  • I'm going to say you answered this, but to be honest we just had to write an R program to do what bibtex does and create a new .bib file that biblatex could read. – Joanna Bryson Aug 2 '19 at 16:42
  • 1
    @JoannaBryson If you think your R script is useful for other people, you might as well write an answer of your own and tell us about the script. Out of curisosity: What exactly do you mean by doing what BibTeX does? Do you mean that you wrote a program that parses .bib files? As far as I understood your main problem is file encoding: You wouldn't have to parse the data structure for that... (It goes without saying that you don't have to accept an answer, so if you feel that your question hasn't been answered properly, you don't need to accept any answer.) – moewe Aug 2 '19 at 16:44
1

Biber by default, uses more strict syntactical validation than BibTeX, and does generally presume that entries are not malformed.

One simple solution may be to use BibTool to pre-process your BibTeX file, to remove unnecessary entries. There are many ways of doing this, but since you said that entries you cite are valid, you can extract only those using your associated aux file:

bibtool -x <file.aux>
  • Ah, and when I properly use the -x command like you suggested, which would only have given well-formed bibtex entries, BibTool fails because not every entry in the aux file is in the .bib file. I normally use bibtex to remind me what citations I still need to look up. :-/ – Joanna Bryson Jun 28 '19 at 18:38
  • 1
    @JoannaBryson Ah, yeah then this might not work so well, unfortunately. You might be able to use other BibTool options or commands to filter for valid entries, but I don't know off-hand. – Coby Viner Jun 28 '19 at 18:49
  • 1
    I've put a note on the BibTool github thingy and emailed one of the biblatex people I happen to know, so in the end you may have the winning answer yet, we'll see. – Joanna Bryson Jun 28 '19 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.