How do I get datatool to play nicely with utf8 inputenc and non-ASCII in the incoming CSV?

Question

If I try to compile the following LaTeX file:

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{datatool}
\begin{document}
\tableofcontents
\appendix
\section{Bill of Materials}

\DTLloadrawdb{bom}{test.csv}
\DTLdisplaydb{bom}
\end{document}

with the following for test.csv:

"name","quantity"
"10μF 16V",5
"470Ω 1/2W",10
"33k×9",1

I get this error:

! Undefined control sequence.
\GenericError  ...                                
                                                    #4  \errhelp \@err@     ...
l.9 \DTLloadrawdb{bom}{test.csv}

How can I fix/work-around this issue?

Answer 1

I find that for UTF-8 input CSV files, it's much simpler to use a UTF-8 aware engine, either LuaLaTeX or XeLaTeX. Either way, you also need to tell datatool how to map the characters if your font doesn't have them.

\documentclass{article}
\usepackage{filecontents}
\begin{filecontents*}{\jobname.csv}
"name","quantity"
"10μF 16V",5
"470Ω 1/2W",10
"33k×9",1
\end{filecontents*}

\usepackage{fontspec}

\usepackage{datatool}
\DTLrawmap{μ}{$\mu$} % add these if your chosen font doesn’t have the character
\DTLrawmap{Ω}{$\Omega$}
\DTLrawmap{×}{$\times$}
\begin{document}
\tableofcontents
\appendix
\section{Bill of Materials}

\DTLloadrawdb{bom}{\jobname.csv}
\DTLdisplaydb{bom}
\end{document}

Answer 2

You can make utf8 characters safe in the edef used by datatool

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{datatool}
\usepackage[T1]{fontenc}
\DeclareUnicodeCharacter{03A9}{\ensuremath{\omega}}
\DeclareUnicodeCharacter{03BC}{\ensuremath{\mu}}
\DeclareUnicodeCharacter{00D7}{\ensuremath{\times}}
\begin{document}
\tableofcontents
\appendix
\section{Bill of Materials}

\makeatletter
\let\zz\UTFviii@two@octets
\def\UTFviii@two@octets#1#2{\noexpand\UTFviii@two@octets\noexpand#1\string#2}
\DTLloadrawdb{bom}{test.csv}
\let\UTFviii@two@octets\zz
\DTLdisplaydb{bom}
\end{document}