Biber 2.11: Data file ... cannot be read in encoding UTF-8 ... ill-formed UTF-8 octet sequence ... at ... Slurper.pm line 63

Question

I have just changed computers, and running Biber 2.11 now produces this error:

Utils.pm:209> ERROR - Data file ... cannot be read in encoding 'UTF-8': 
Can't decode ill-formed UTF-8 octet sequence <96> at C:\ ... Slurper.pm line 63

My .bib file is huge, having accumulated over years. Thus, I suspect that there's something malformed in it. Is there any rapid way to trouble-shoot it?

The new machine runs Win 10, Biber 2.11, XeLaTeX 2018.10.31, LaTeX2e 2018-04-01 and TeXLive 2018. The old machine ran Win 7 and slightly earlier versions of all of the above.

A MWE of the driver .tex file:

\documentclass{article} % also with beamer

\usepackage[utf8]{inputenc} % problem arises whether or not include this
\usepackage[backend=biber,bibencoding=utf8]{biblatex} %  problem arises whether or not include this
\addbibresource{bib-mwe.bib}

\begin{document}

cite reference \cite{mal-rwws}

\printbibliography

\end{document} 

Thanks!

Answer 1

As explained in https://github.com/plk/biber/issues/226 the underlying library that Biber uses to read .bib files does not provide a line number for the error. So Biber can't tell you the place of the error.

Usually the problem is simply that the entire file does not have the guessed/declared encoding. Note that the entire file content matters, so even the comments could be an issue.

For the MWE the .bib file must be encoded in UTF-8 (or a compatible encoding like US-ASCII). Even without the explicit bibencoding=utf8, biblatex would guess from the \usepackage[utf8]{inputenc} that you are using UTF-8, and if your LaTeX is newer than 2018-04-01, UTF-8 would even be assumed without inputenc since it is the standard now.

If a .bib file acts up, you should go through the following steps.

1. Determine which encoding it should have (usually UTF-8 is a good idea, but if you have declared a different encoding in your .tex file, biblatex will normally assume that your .bib file uses the same encoding, of course that can be overridden with bibencoding).
2. Find out which encoding your .bib file actually has.
3. If the encodings don't match, you need to recode your .bib file or tell biblatex to try a different encoding.

It may work to just copy and paste the contents of your .bib file to a new file that is explicitly set up as UTF-8.

Answer 2

I ran into the same problem. Instead of using a text editor, I opened the .bib file in jabref. You can change the library encoding using 'Library Properties' from the 'File' menu. Then save the library. This also changed the encoding specification on the first line of the .bib file, which I would otherwise have had to change manually (when I would have used the text editor).

Answer 3

Thanks for the helpful comments.

Following David's, the problem is solved by:

1. open the .bib file in a text editor (e.g., Notepad)
2. set the encoding in WinEdt (my LaTeX editor) to UTF-8 by clicking on the menu bar at the bottom (originally set to ANSI)
3. create a blank document in WinEdt
4. paste the contents of the original .bib file from Notepad into WinEdt
5. save the new WinEdt document as a .bib file
6. compile using the new .bib file