# Can't use utf8 encoding with datatool

I am using datatool to manage tables and I am getting errors when the data.csv file (encoded either as UTF-8 or ANSI) has characters like 'á'. I can get it around by having data.csv encoded as ANSI and using {\'a}, but that is too boring to do. The interesting thing is, if I add \usepackage[T1]{fontenc}, I can encode data.csv as UTF-8 and use chars like 'á' without problems, but the font looks damn ugly.

file.tex

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{datatool}
\begin{document}
The quick brown fox jumps over the lazy dog
á é õ ç {\c{c}} {\~a}
\begin{table}
\DTLdisplaydb{data}
\end{table}
\end{document}


data.csv

a,e
a,c


data.csv

{'/a},{'/e}
{\~a},{\c{c}}


data.csv

á,é
ã,ç

-
Could you please add a sample of data.csv? –  egreg Oct 28 '12 at 18:49
Using \DTLloaddb[headers={á,é}]{data}{data.csv} and setting the first line of data.csv to , (yes, one comma) loads everything just fine. –  Qrrbrbirlbel Oct 28 '12 at 19:10
wow weirdly that works –  Fernando Antonio Nogueira Oct 28 '12 at 19:12
@Qrrbrbirlbel's fix works! sneaky edit... –  Silex Oct 28 '12 at 19:15
@Silex re-edit ;) @Fernando This seems to be only an issue with the headers (i.e. the first line of the .csv) where datatool seems to applying some TeX-foo that can't handle accented characters. –  Qrrbrbirlbel Oct 28 '12 at 19:19

I see the following solutions:

1. From my comment:

• Replace the first line of your .csv with: ,
• Add the headings to the \DTLloaddb command:

\DTLloaddb[headers={á,é}]{data}{data.csv}

2. Use lmodern with the T1 font encoding. (recommended)

3. Taking a look in T1.enca) we can find the following lines:

\DeclareTextAccent{\'}{T1}{1}
% … and much later …
\DeclareTextComposite{\'}{T1}{a}{225}
\DeclareTextComposite{\'}{T1}{e}{233}


If we try to copy this for the OT1 encoding we don't get any single char (but it compiles!), because there isn't any glyph at position 225 (E116) or 233 (E916) in the OT1 font encoding but they are in fact in the T1 font encoding.

What TeX actually does when you use \'e:
It sets the special accent glyph at 1316 (1910/´) over 6516 (10110/e). Try it yourself:

\accent19 e


As TeX knows that this is a special accent glyph it can set it correct over capital letters. But let's let the TeXbookb) (which didn't exactly used OT1 but a font encoding with “quite” the same layoutc)) speak for itself:

Appendix B shows that plain TeX handles most of the accents by using TeX's \accent primitive. For example, \'#1 is equivalent to {\accent19 #1}, where #1 is the argument being accented. The general rule is that \accent<number> puts an accent over the next character; the <number> tells where that accent appears in the current font.

And regarding capital letters:

The accent is assumed to be properly positioned for a character whose height equals the x-height of the current font; taller or shorter characters cause the accent to be raised or lowered, taking due account of the slantedness of the fonts of accenter and accentee.

Line 696 of plain.tex looks like this:

\def\'#1{{\accent19 #1}}


a) texmf/tex/latex/base/T1.enc b) texmf/doc/knuth/texbook.tex.gz
c) One can read on that more in texmf/doc/latex/base/encguide.pdf:

Little attention was paid to font encodings prior to the arrival of TeX 3. Up to that time, one used Donald Knuth’s fonts (the Computer Modern family, using the encodings we now refer to as OT1 and the OM series), or one was on one’s own.

The Computer Modern text encoding raises problems in unmodiﬁed TeX, because hyphenation cannot break words containing \accent commands. Even in those Western European languages for which the OT1 encoding has symbols for the necessary \accent-based diacritics, this shortcoming ruins typesetting of running text.

-
The guess is almost wrong. :) What LaTeX does with \'e does not depend on UTF-8 being used or not; if T1 is the font encoding the instruction \'e is translated into the command "print the character at position 233 in the current font"; with OT1 it resolves to \accent19 e. –  egreg Oct 28 '12 at 20:03
@egreg Better now? –  Qrrbrbirlbel Oct 28 '12 at 20:21
Very good; I'd add only that OT1 encoded fonts (in LaTeX-speak) use (quite) the same layout as the one explained in the TeXbook. –  egreg Oct 28 '12 at 20:26

Using fontenc and encoding your data as UTF-8 seems like your best bet. If you don't like the font you can load a package with a font you do like.

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{datatool,tgbonum}
\begin{document}
The quick brown fox jumps over the lazy dog
á é õ ç {\c{c}} {\~a}
\begin{table}