# How do I get datatool to play nicely with utf8 inputenc and non-ASCII in the incoming CSV?

If I try to compile the following LaTeX file:

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{datatool}
\begin{document}
\tableofcontents
\appendix
\section{Bill of Materials}

\DTLdisplaydb{bom}
\end{document}


with the following for test.csv:

"name","quantity"
"10μF 16V",5
"470Ω 1/2W",10
"33k×9",1


I get this error:

! Undefined control sequence.
\GenericError  ...
#4  \errhelp \@err@     ...


How can I fix/work-around this issue?

I find that for UTF-8 input CSV files, it's much simpler to use a UTF-8 aware engine, either LuaLaTeX or XeLaTeX. Either way, you also need to tell datatool how to map the characters if your font doesn't have them.

\documentclass{article}
\usepackage{filecontents}
\begin{filecontents*}{\jobname.csv}
"name","quantity"
"10μF 16V",5
"470Ω 1/2W",10
"33k×9",1
\end{filecontents*}

\usepackage{fontspec}

\usepackage{datatool}
\DTLrawmap{μ}{$\mu$} % add these if your chosen font doesn’t have the character
\DTLrawmap{Ω}{$\Omega$}
\DTLrawmap{×}{$\times$}
\begin{document}
\tableofcontents
\appendix
\section{Bill of Materials}

\DTLdisplaydb{bom}
\end{document}


• I went with the solution of a UTF-8 aware engine in the end as it's the simplest way to deal with these sorts of issues. I may have to adopt LuaTeX as my standard engine from here on out even :) – ThreePhaseEel Oct 21 '17 at 3:47

You can make utf8 characters safe in the edef used by datatool

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{datatool}
\usepackage[T1]{fontenc}
\DeclareUnicodeCharacter{03A9}{\ensuremath{\omega}}
\DeclareUnicodeCharacter{03BC}{\ensuremath{\mu}}
\DeclareUnicodeCharacter{00D7}{\ensuremath{\times}}
\begin{document}
\tableofcontents
\appendix
\section{Bill of Materials}

\makeatletter
\let\zz\UTFviii@two@octets
\def\UTFviii@two@octets#1#2{\noexpand\UTFviii@two@octets\noexpand#1\string#2}
\let\UTFviii@two@octets\zz
\DTLdisplaydb{bom}
\end{document}

• Alternatively use \DTLloaddb instead of \DTLloadrawdb to avoid the edef. – Nicola Talbot Oct 8 '17 at 11:56
• @NicolaTalbot but that would require reading the manual rather than running \tracingall and seeing where it fell over. So is out of scope for this site:-) – David Carlisle Oct 8 '17 at 11:57
• :-) Actually it also seems to be working with \DTLloadrawdb provided the Unicode characters are declared. I could change the \xdef to \protected@xdef within \DTLloadrawdb, but that command's not intended for CSV files that contain macros and special characters are supposed to be mapped when using it. I'm not sure if the change to \protected@xdef would cause any unwanted side-effects. – Nicola Talbot Oct 8 '17 at 12:16
• @NicolaTalbot we are considering using \protected\edef for the undefined character error (or possibly the defined characters as well) that would make them safe in an edef automatically, but is etex so post 1990 so a bit modern for us. – David Carlisle Oct 8 '17 at 12:19