4

I just updated my MikTeX distribution and am now running biber 2.15 with biblatex 3.15a. And all of a sudden, I run into problems. I have a bib-file with 1888 entries, and trying to use that now seems to break biblatex, although it worked fine on earlier versions. I do not know if the problem lies with biber or biblatex, but try the following:

I created two bib-files out of my original one, available here and here. With the following MWE it works fine and both citations are resolved:

\documentclass{article}
\usepackage[french, german, UKenglish]{babel}
\usepackage[backend=biber, style=authoryear]{biblatex}
\addbibresource{Test.bib}
\addbibresource{Test2.bib}

\begin{document}
Let's test: \textcite{Arnold2002}, \textcite{Setty1973}

\printbibliography
\end{document}

However, when I combine both bib-files into one and try to compile with that, none of the citations is resolved. Since it works fine when having the database split in two, I cannot see any other reason for this behaviour other than the new biber or biblatex having a limit of file-size/entry-number in the bib-file. Not the amount of entries actually cited, mind you (as it was in bibtex), but the sheer number of entries in the database. Or do I overlook something? Any help would be greatly appreciated.

EDIT 1:

Here is the .blg-file. Indeed, there is an error message "[1095]", but I cannot make sense of it:

[0] Config.pm:312> INFO - This is Biber 2.15 (beta)
[0] Config.pm:315> INFO - Logfile is 'DatabaseTest.blg'
[311] biber-MSWIN64:330> INFO - === 
[385] Biber.pm:415> INFO - Reading 'DatabaseTest.bcf'
[882] Biber.pm:952> INFO - Found 2 citekeys in bib section 0
[998] Biber.pm:4334> INFO - Processing section 0
[1068] Biber.pm:4523> INFO - Looking for bibtex format file 'TestFull.bib' for section 0
[1095] Utils.pm:411> ERROR - Data file 'TestFull.bib' cannot be read in encoding 'utf8': Wide character at C:\Users\Manue\AppData\Local\Temp\par-4d616e7565\cache-b32225defac3568d1e921af5a7245ca571b236aa\inc\lib/Win32/Unicode/File.pm line 163.
[1095] Biber.pm:132> INFO - ERRORS: 1

EDIT 2:

I now identified the limit length. The file available here contains as many entries as I manage to process. It contains 1477 entries, 25026 lines, and 1,048,339 bytes. This file just works, but as soon as I add just one more mock entry like

@Article{Test3,
  author       = {Author, A. and Author B. and Author, C.},
  date         = {2020},
  journaltitle = {A Journal},
  title        = {A nice title of a certain topic which I cannot even imagine},
  issue        = {3},
  pages        = {1--18},
  volume       = {12}
}

I get the same error as above. This seems to be the length limit with which biblatex works properly.

EDIT 3:

Here is the full file that does not work directly from JabRef (as .bib) without any changes. And here is the same file as hex dump exported with Cygwin.

12
  • This works fine for me (with a file made as cat Test.bib Test2.bib >combined.bib). Even though your bib files are a bit strange in that their last lines aren't terminated so the entry Setty1973 doesn't begin on a new line in the combined version even those entries work fine, and the only warning I get from biber is about a repeated entry that hasn't to do with this. Have you seen if you get any error messages or warnings from biber?
    – pst
    Sep 14, 2020 at 10:55
  • @DavidCarlisle Thank you for the suggestion. Indeed, there is an error message in the .blg.-file: ERROR - Data file 'TestFull.bib' cannot be read in encoding 'utf8': Wide character at C:\Users\Manue\AppData\Local\Temp\par-4d616e7565\cache-b32225defac3568d1e921af5a7245ca571b236aa\inc\lib/Win32/Unicode/File.pm line 163.. I cannot say I can really make sense of that, other that something in the file is not to biblatex liking in utf8 encoding... Sep 14, 2020 at 11:41
  • @DavidCarlisle But this does not make sense for a couple of reasons: (1) The original file is directly copied from JabRef, where it is stored in UTF8 encoding, which is also indicated in the file itself. (2) The two separate files for download are exactly the same file, split in Notepad++ without any encoding change, and they work. Notepad++ confirms that they are UTF8-encoded. (3) When combining these two, that did work, into one file in Notepad++, so it is essentially the same file, it stops working. And the combined file, as Notepad++ confirms, is again in UTF8-encoding. Sep 14, 2020 at 11:55
  • Whatever was in that (1) you started with is not relevant, since all the difference is between (2) the two files that you have uploaded to dropbox which work fine and (3) your bigger file that you made with some join operation in Notepad++. If you upload your bigger file as well we can probably see what went wrong in that join operation, but it is even better if you as David suggested edit down your longer file down to just one entry which still shows the problem.
    – pst
    Sep 14, 2020 at 12:55
  • @pst This is a misunderstanding. I started off with the full file, and it did not work, although it always did before the update. I then started reducing the full file to find in which entry the error lies. Surprisingly, I quickly realized that it worked perfectly fine as soon as I split the sample into two and included both without changing anything about the entries themselves, making it practically impossible that it can be a problem with the entries themselves. I now added under EDIT 2 a file of exactly the length that just works. If I add one more, simple and madeup entry, it breaks. Sep 14, 2020 at 13:10

5 Answers 5

9

According to new trials by Ulrike Fischer, it is now reasonably certain that this is a bug in the new version of biber, which breaks at a certain bib-file size:

https://github.com/plk/biber/issues/332

EDIT:

For the time being, there are two possible fixes

  1. Split your bib-file into several files, each with a size <1MB
  2. Use the current development version 2.16 of biber, available on sourceforge, where the bug was fixed.

EDIT 2:

biber 2.16 was released in April 2021. bib-files of up to 100MB should now be supported. If you experience this issue on your system, make sure to use biber 2.16 or higher.

2

I cannot write comments (yet) so as an answer: This sounds like a OS-specific problem. I tested on Linux without an issue. There is a very recent issue reported on the github issue tracker (https://github.com/plk/biber/issues/332) similar to your problem and it may be related to EOL markers. Could you check if the solution in the issue tracker works for you? In particular, the user there reported that the conversion to LF-only EOL markers solved the problem.

6
  • The working files also have CRLF. I think it's more likely that what happened there was that when the user re-saved the file their editor also fixed some encoding problem.
    – pst
    Sep 14, 2020 at 13:30
  • Many thanks, this seems like exactly this problem and as if it is Windows-specific. Sep 14, 2020 at 13:44
  • @pst This might be, but I raise that neither the original file from JabRef, nor the reassambled file from Notepad++ (after first splitting the file in two in Notepad++ as well) do work. So I still do not think that it is an encoding problem. Also because, again, it did work with exactly the same file before the update. Sep 14, 2020 at 13:49
  • Unfortunately, the approach suggested by nussaibahrs on GitHUB did not work in my case, although all Windows-line endings have been recoded to Linux-like LF. Sep 14, 2020 at 13:59
  • 1
    I don't think it is an encoding issue, unitools on Linux does not show anything unusual at the very least. It is odd that it happens near the 1MiB line and the github user may have been lucky that CRLF to LF conversion reduced the file size enough to bring it below the 1MiB. I originally thought it may be the size of the buffer and that the CRLF got split incorrectly at the end of a read and the conversion might work. You may need to keep your files below 1MiB for now as a workaround or use the dev version as per the github issue. Sep 14, 2020 at 23:57
0

I followed this to basically make sure my EOL was LF rather than CRLF and this sorted out my issues as well as making sure my file was encoded as UTF-8.

2
  • Thank you. Yes, I did that. It did not help in my case. This seems to be a very peculiar bug. Sep 14, 2020 at 15:15
  • Look like I spoke too fast. I got the error again as soon as I added new references to the file, which seems to indicate a size/length specific issue. I have now resorted to using multiple bib files instead of one main one. Sep 14, 2020 at 16:51
0

One of our students ran into the same problem, but it couldn't be solved simply by splitting the .bib file into chunks or deleting large parts of it that weren't really needed. It seems like the most pressing issues were the @MvBook coded entries in the .bib file (also done in JabRef like yours) as well as references to them in the text by /volcite (and \tvolcite etc.). After changing all @MvBook-s to @Book-s and all /volcite-s to /cite-s, the file once more runs through.

2
  • 1
    Would it be possible to reproduce the problem and post the problematic .bib file to the Biber issue tracker (github.com/plk/biber/issues)? If the problem is not resolved by bringing the file size down to below 1MB, then it might be a different issue that hasn't yet been solved. It certainly sounds odd that the entry type should be the problem.
    – moewe
    Sep 25, 2020 at 15:45
  • 1
    Come to think of it a problem with \volcite sounds sounds like it could be relate to github.com/plk/biblatex/issues/1039 (the bug will be fixed in the next biblatex release, a workaround is available).
    – moewe
    Sep 26, 2020 at 5:44
0

I have a bib file with more than 4500 entries that works perfectly with TexLive2019 and TexStudio 3.0.1 under Windows 10. When I moved to a different Windows 10 computer, I downloaded TexLive2020 and TexStudio 3.0.1. I immediately got this error when attempting to compile my document. It is the same bib file running on both machines because I download it from Google Drive. So this definitely appears to be a bug rather than a file encoding problem.

EDIT: I used Notepad++ and JABREF to split the file into three .bib files and then used a \addbibresource{} for each part in TexStudio. (I could not get it to recognize the three filenames in a single add command with no spaces and a comma between files.) Now it runs as it should.

2
  • 1
    Yes, it has been established that this is a bug and the two possible fixes are listed in the accepted answer! Sep 26, 2020 at 13:49
  • Hello newcomers. This sounds like an elaborated comment but not an answer. Please be informed that non-answers in the answer section will be deleted. (That said, I don't know if this one counts. I'll let others decide.)
    – Symbol 1
    Sep 26, 2020 at 17:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .