Datatool, UTF8 and Export to a file

Question

The title contains the problem: I have a datatool table, which contain accented characters (öäü) and I want to export the table into a CSV file :

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{datatool}

\begin{document}
\DTLnewdb{table}
\DTLnewrow{table}
\DTLnewdbentry{table}{name}{aouäöü}


\DTLdisplaydb{table}
\DTLsavedb{table}{export.csv}

\end{document}

With the package inputenc, the display of the table is ok:

name
aouäöü

But in the created file export.csv there is :

name
aou\IeC {\"a}\IeC {\"o}\IeC {\"u}

The best for me would be if the exported file was UTF8 encoded (instead of Ascii), and if it contained aouäöü. A second class solution which I could accept, would be, if the exported file contained

aou\"a\"o\"u

I could also accept a macro \convert which would take öäü as argument and would return \"a\"o\"u, but I don't know if it exists.

Answer 1

You can get the \"a version by redefining \IeC:

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{datatool}

\begin{document}
\DTLnewdb{table}
\DTLnewrow{table}
\DTLnewdbentry{table}{name}{aouäöü}


\DTLdisplaydb{table}
\def\IeC#1{#1}
\DTLsavedb{table}{export.csv}

\end{document}

For real UTF8 you could try something like this: Complete (!) (the first entry is 128, the last 255, the second argument contains the number in hex) and then save this as identity.def:

\ProvidesFile{identity.def}
   [2012/02/23 v1.0 Input encoding file]
\makeatletter
\DeclareInputText{128}{\string^^80}
%....
\DeclareInputText{164}{\string^^a4}
%....
\DeclareInputText{182}{\string^^b6}
%....
\DeclareInputText{188}{\string^^bc}
%....
\DeclareInputText{195}{\string^^c3}
% ...
\DeclareInputText{228}{\string^^e4}
%...
\DeclareInputText{246}{\string^^f6}
%...
\DeclareInputText{252}{\string^^fc}
%...
\makeatother
\endinput

Then this (I hope without side-effects ;-)) should give you UTF8 export (if the main document is UTF8 encoded. If it is in e.g. ansinew, the export will be ansinew too):

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{datatool}

\begin{document}
\DTLnewdb{table}
\DTLnewrow{table}
\DTLnewdbentry{table}{name}{aouäöü}


\DTLdisplaydb{table}

{\inputencoding{identity}
\DTLsavedb{table}{export.csv}
}

öäü
\end{document}

Answer 2

A patch that might accomplish the \"a version is

\usepackage{etoolbox}
\makeatletter
\patchcmd{\DTLsavedb}
  {\protected@write\@dtl@write{}{\@dtl@row}}
  {{\let\IeC\@firstofone\protected@write\@dtl@write{}{\@dtl@row}}}
  {}{}
\makeatother

but this can fail for other accented characters. The problem is in how \@dtl@row is defined (with \protected@edef) and the implementation of UTF-8 support in LaTeX that guarantees correct writes as far as LaTeX is concerned, but is not what one can expect for other purposes like yours.

EDIT: Expanding on Ulrike's idea, here's a version that should be safe.

This code should go in the preamble

\makeatletter
\newcommand{\DTLsavedbutf}[2]{%
  \begingroup\count@=127
  \loop\ifnum\count@<\@cclv
    \advance\count@\@ne
    \begingroup\lccode`!=\count@
    \lowercase{\endgroup
      \expandafter\DeclareInputText\expandafter{\number\count@}{\string!}}
  \repeat
  \DTLsavedb{#1}{#2}\endgroup}
\makeatother

and then the command \DTLsavedbutf will write all characters (with high bit set) without any special interpretation.

If you want to apply this to every \DTLsavedb command, then the code should redefine this command:

\makeatletter
\let\original@DTLsavedb\DTLsavedb
\renewcommand{\DTLsavedb}[2]{%
  \begingroup\count@=127
  \loop\ifnum\count@<\@cclv
    \advance\count@\@ne
    \begingroup\lccode`!=\count@
    \lowercase{\endgroup
      \expandafter\DeclareInputText\expandafter{\number\count@}{\string!}}
  \repeat
  \original@DTLsavedb{#1}{#2}\endgroup}
\makeatother

In this way you can use \DTLsavedb and not a separate command.