Datatool, UTF8 and Export to a file

The title contains the problem: I have a datatool table, which contain accented characters (öäü) and I want to export the table into a CSV file :

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{datatool}

\begin{document}
\DTLnewdb{table}
\DTLnewrow{table}
\DTLnewdbentry{table}{name}{aouäöü}

\DTLdisplaydb{table}
\DTLsavedb{table}{export.csv}

\end{document}


With the package inputenc, the display of the table is ok:

name
aouäöü


But in the created file export.csv there is :

name
aou\IeC {\"a}\IeC {\"o}\IeC {\"u}


The best for me would be if the exported file was UTF8 encoded (instead of Ascii), and if it contained aouäöü. A second class solution which I could accept, would be, if the exported file contained

aou\"a\"o\"u


I could also accept a macro \convert which would take öäü as argument and would return \"a\"o\"u, but I don't know if it exists.

-
Just to note, this works out of box in XeLaTeX and LuaLaTeX. – Andrey Vihrov Feb 23 '12 at 8:27
ok. I have to admit, i don't know them. I use texlive. What would i have to do in order to use them ? A complete installation of a new latex distribution ? Would i have to change something in all my old .tex documents ? – nicolas roy Feb 23 '12 at 9:58
TeX Live already has them. Search this website for xetex and luatex. Amongst other stuff you will find information on how to migrate from pdfTeX. – Andrey Vihrov Feb 23 '12 at 10:00
these are replacements of the pdflatex command, do i understand correctly ? – nicolas roy Feb 23 '12 at 10:05
pdfTeX, XeTeX, LuaTeX are TeX engines, pdflatex, xelatex and lualatex is how you invoke these engines with the LaTeX format. – Andrey Vihrov Feb 23 '12 at 10:07

2 Answers

You can get the \"a version by redefining \IeC:

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{datatool}

\begin{document}
\DTLnewdb{table}
\DTLnewrow{table}
\DTLnewdbentry{table}{name}{aouäöü}

\DTLdisplaydb{table}
\def\IeC#1{#1}
\DTLsavedb{table}{export.csv}

\end{document}


For real UTF8 you could try something like this: Complete (!) (the first entry is 128, the last 255, the second argument contains the number in hex) and then save this as identity.def:

\ProvidesFile{identity.def}
[2012/02/23 v1.0 Input encoding file]
\makeatletter
\DeclareInputText{128}{\string^^80}
%....
\DeclareInputText{164}{\string^^a4}
%....
\DeclareInputText{182}{\string^^b6}
%....
\DeclareInputText{188}{\string^^bc}
%....
\DeclareInputText{195}{\string^^c3}
% ...
\DeclareInputText{228}{\string^^e4}
%...
\DeclareInputText{246}{\string^^f6}
%...
\DeclareInputText{252}{\string^^fc}
%...
\makeatother
\endinput


Then this (I hope without side-effects ;-)) should give you UTF8 export (if the main document is UTF8 encoded. If it is in e.g. ansinew, the export will be ansinew too):

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{datatool}

\begin{document}
\DTLnewdb{table}
\DTLnewrow{table}
\DTLnewdbentry{table}{name}{aouäöü}

\DTLdisplaydb{table}

{\inputencoding{identity}
\DTLsavedb{table}{export.csv}
}

öäü
\end{document}

-

A patch that might accomplish the \"a version is

\usepackage{etoolbox}
\makeatletter
\patchcmd{\DTLsavedb}
{\protected@write\@dtl@write{}{\@dtl@row}}
{{\let\IeC\@firstofone\protected@write\@dtl@write{}{\@dtl@row}}}
{}{}
\makeatother


but this can fail for other accented characters. The problem is in how \@dtl@row is defined (with \protected@edef) and the implementation of UTF-8 support in LaTeX that guarantees correct writes as far as LaTeX is concerned, but is not what one can expect for other purposes like yours.

EDIT: Expanding on Ulrike's idea, here's a version that should be safe.

This code should go in the preamble

\makeatletter
\newcommand{\DTLsavedbutf}[2]{%
\begingroup\count@=127
\loop\ifnum\count@<\@cclv
\advance\count@\@ne
\begingroup\lccode!=\count@
\lowercase{\endgroup
\expandafter\DeclareInputText\expandafter{\number\count@}{\string!}}
\repeat
\DTLsavedb{#1}{#2}\endgroup}
\makeatother


and then the command \DTLsavedbutf will write all characters (with high bit set) without any special interpretation.

If you want to apply this to every \DTLsavedb command, then the code should redefine this command:

\makeatletter
\let\original@DTLsavedb\DTLsavedb
\renewcommand{\DTLsavedb}[2]{%
\begingroup\count@=127
\loop\ifnum\count@<\@cclv
\advance\count@\@ne
\begingroup\lccode!=\count@
\lowercase{\endgroup
\expandafter\DeclareInputText\expandafter{\number\count@}{\string!}}
\repeat
\original@DTLsavedb{#1}{#2}\endgroup}
\makeatother


In this way you can use \DTLsavedb and not a separate command.

-
thanks for the patch. By the way, is datatool the only one (or at least the canonical one) package, that allows one to store and read values from tables ? – nicolas roy Feb 23 '12 at 9:59
yes ... it seems to work ... but it's almost black magic for me ! :) – nicolas roy Feb 23 '12 at 15:13
why couldn t we put the whole code (without the \DTLsavedb{#1}{#2} in a separate command and then call the usual \DTLsavedb{table}{file.csv} ? because it could cause side effects on the rest of the document ? – nicolas roy Feb 23 '12 at 15:17
well i meant : what is against defining a command which only does all the \DeclareInputText, and using the original DTLsavedb ? – nicolas roy Feb 23 '12 at 17:10
You have to do those \DeclareInputText only for that specific purpose, otherwise printing UTF-8 characters in the file would be disabled. So enclosing the call of \DTLsavedb in a group is necessary. – egreg Feb 23 '12 at 17:13