# Can't replace non-ascii characters with biblatex

This is a follow-up to my question Translate content in fields in bibliography entries (biblatex), which revealed a problem when trying to replace content in a .bib file when that content contain non-ascii characters like æøå.

Compiling the following MWE works fine. In this case, the string def is replaced by abc:

\documentclass{article}
\usepackage[style = authoryear-comp]{biblatex}
\usepackage{filecontents}
\begin{filecontents}{\jobname.bib}
@BOOK{lennon1968,
AUTHOR = "John Lennon",
TITLE = "def",
YEAR = "1968"}
\end{filecontents}
\DeclareSourcemap{
\maps[datatype = bibtex]{
\map{
\step[fieldsource = title,
match = {def},
replace = {abc}]
}
}
}
\begin{document}
\nocite{*}
\printbibliography
\end{document}


But when the string contains non-ascii characters like æøå, no output is produced with LaTeX:

\documentclass{article}
\usepackage[style = authoryear-comp]{biblatex}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{filecontents}
\begin{filecontents}{\jobname.bib}
@BOOK{lennon1968,
AUTHOR = "John Lennon",
TITLE = "æøå",
YEAR = "1968"}
\end{filecontents}
\DeclareSourcemap{
\maps[datatype = bibtex]{
\map{
\step[fieldsource = title,
match = {æøå},
replace = {abc}]
}
}
}
\begin{document}
\nocite{*}
\printbibliography
\end{document}


The console produces this output (biblatextest64.tex being the name of the document) :

Failure to make 'biblatextest64.pdf'
Collected error summary (may duplicate other messages):
biber biblatextest64: Command for 'biber biblatextest64' gave return code 256
Latexmk: Use the -f option to force complete processing,
unless error was exceeding maximum runs of latex/pdflatex.
Latexmk: Errors, so I did not complete making targets
C:\texlive2013\bin\win32\runscript.tlu:650: command failed with exit code 12:
perl.exe c:/texlive2013/texmf-dist/scripts/latexmk/latexmk.pl -pdf biblatextest64.tex


A pdf is produced when compiling with XeLaTeX, but no substitution has taken place:

\documentclass{article}
\usepackage[style = authoryear-comp]{biblatex}
\usepackage{fontspec}
\usepackage{filecontents}
\begin{filecontents}{\jobname.bib}
@BOOK{lennon1968,
AUTHOR = "John Lennon",
TITLE = "æøå",
YEAR = "1968"}
\end{filecontents}
\DeclareSourcemap{
\maps[datatype = bibtex]{
\map{
\step[fieldsource = title,
match = {æøå},
replace = {abc}]
}
}
}
\begin{document}
\nocite{*}
\printbibliography
\end{document}


-
Move the \DeclareSourcemap before inputenc. –  Ulrike Fischer Apr 30 at 11:48
Sorry I confuse match and replace. Moving \DeclareSoureMap gets rid off the errors but the match is not replaced. –  Ulrike Fischer Apr 30 at 12:02
Did you try match = \regexp{æøå} - this is used to protect special chars in matches. –  PLK May 3 at 12:34
@PLK I tried it now, still no substitution is made. –  Sverre May 3 at 12:39
This was a bug with a NFD unicode boundary. It should be fixed in 1.9 DEV version on SF now (you need to be using biblatex 2.9 with biber 1.9 which is also on SF). –  PLK May 3 at 13:42

It seems that the value given to match is passed through TeX's expansion mechanism, which is a big problem, but also a help, since we can (also selectively) use \detokenize.

\documentclass{article}
\usepackage[style = authoryear-comp]{biblatex}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

\usepackage{filecontents}
\begin{filecontents}{\jobname.bib}
@BOOK{lennon1968,
AUTHOR = "John Lennon",
TITLE = "æøå",
YEAR = "1968"}
\end{filecontents}

\DeclareSourcemap{
\maps[datatype = bibtex]{
\map{
\step[fieldsource = title,
match = {\detokenize{æøå}},
replace = {abc}]
}
}
}
\begin{document}
\nocite{*}
\printbibliography
\end{document}


This is what is written in the .bcf file:

  <!-- SOURCEMAP -->
<bcf:sourcemap>
<bcf:maps datatype="bibtex" level="user">
<bcf:map>
<bcf:map_step map_field_source="title" map_match="æøå" map_replace="abc"/>
</bcf:map>
</bcf:maps>


Without \detokenize, the result is

  <!-- SOURCEMAP -->
<bcf:sourcemap>
<bcf:maps datatype="bibtex" level="user">
<bcf:map>
<bcf:map_step map_field_source="title" map_match="\T1\ae \T1\o <E5>" map_replace="abc"/>
</bcf:map>
</bcf:maps>


So it seems that \detokenize works correctly. However, the replacement doesn't take place, which seems to be a problem with Biber.

To wit, I tried the following Perl script (adapted from http://stackoverflow.com/a/1572156)

use warnings;
use strict;
use utf8;
binmode STDOUT, "utf8";
my $string = "æøå";$string =~ s/æøå/abc/gi;

print "\$string\n";


and the result on the terminal was

abc

-
I'm with you until your Perl script. I see that your Perl script works, but how does that relate to biber and the fact that it doesn't replace the characters? –  Sverre Apr 30 at 12:19
@Sverre Biber is programmed in Perl. –  egreg Apr 30 at 12:23
Does that mean there is a solution to this ... ? –  Sverre Apr 30 at 12:23
@Sverre Not from me, but probably from Biber's developers. –  egreg Apr 30 at 12:24
In other words a bug report should be made? Presumably here: github.com/plk/biber/issues? I'd recommend that someone who knows what they're talking about (i.e. you) submit it rather than me (I would do little more than link to your answer here). –  Sverre Apr 30 at 12:26