hyperxmp + xelatex bug: duplication of pdfkeywords

Question

I found an annoying bug(?) relating to the pdfkeywords metadata field which hyperref allows to set. When hyperref is used in combination with hyperxmp and the document is compiled with xelatex the keywords are duplicated in the pdf metadata (both in the regular pdf properties and the XMP metadata).

Here's an MWO:

\documentclass{article}

\usepackage[xetex]{hyperref}% remove "xetex" parameter when compiling with pdflatex instead of xelatex
\hypersetup{
    pdfauthor = {Matthias},
    pdftitle = {MWO},
    pdfkeywords = {Test, MWO, This is a test, Lipsum}
}
\usepackage{hyperxmp}

\usepackage{lipsum}

\begin{document}
\lipsum[1-3]
\end{document}

When I compile this with pdflatex (remove the xetex parameter in \usepackage{hyperref}) I get:

• in the keywords field of the PDF properties:

  Test, MWO, This is a test, Lipsum

• in the keywords field of the XMP metadata:

  Test; MWO; This is a test; Lipsum

This is the intended behaviour (hyperxmp replaces commas with ; in the XMP field, cf. hyperxmp documentation).

However, when compiling the MWO with xelatex I get:

• in the keywords field of the PDF properties:

  "Test, MWO, This is a test, Lipsum"; Test;  MWO;  This is a test;  Lipsum

• in the keywords field of the XMP metadata:

  "Test, MWO, This is a test, Lipsum"; Test; MWO; This is a test; Lipsum

As you can see the keywords are repeated. First the ,-separated is put between "" (and treated as a single keyword) and then the ;-separated sequence follows.

I have tried changing the package loading order and compiling with xelatex without specifying the xetex parameter on \usepackage{hyperref}, all to no avail. Any ideas on how to fix or work around this?

Answer 1

Package hyperxmp in version 2011/06/12 v1.4 does not use UTF-8 for the XMP data. Instead the data bytes are given in PDFDocEncoding or even UTF-16BE. Also its trimspaces macro breaks because of this. The BOM (byte order mark) is not written correctly in case XeTeX, ...

Strings as result of \pdfstringdef (\@pdfauthor and friends) can have different encodings dependent of option settings and drivers. \ifHy@unicode is not useful for the encoding detection. With pdfencoding=auto the string can also have PDFDocEncoding! Also there is an exception for XeTeX, where the string might have big chars (with character code > 127). XeTeX unlike all other drivers only provide a very ugly unspecified and incomplete interface if an application wants to write arbitrary binary data.

The string detection can be done by looking for the BOM that \pdfstringdef sets for Unicode strings. Additionally (for XeTeX) the string needs to be checked for big chars.

In the next step the XMP data are written. Package hyperxmp uses UTF-8. Therefore the patch below encodes the string in UTF-8 with XML escapings. (Using the original PDFDocEncoding or Unicode are wrong, both encodings differ to UTF-8.)

The patch also fixes the BOM marker for XeTeX and LuaTeX.

\documentclass{article}

\usepackage[
  pdfencoding=auto
%  unicode,
]{hyperref}
\hypersetup{
    pdfauthor = {Matthias<abc>},
    pdftitle = {MWO\textalpha\textomega},
    pdfkeywords = {Test, MWO, This is a test, Lipsum}
}

\usepackage{hyperxmp}
\usepackage{ltxcmds}[2010/04/26]
\usepackage{ifxetex}
\usepackage{ifluatex}

\makeatletter
\newif\ifhyxmp@unicodetex
\ifluatex
  \hyxmp@unicodetextrue
\else\ifxetex
  \hyxmp@unicodetextrue
\fi\fi

\renewcommand*{\hyxmp@xmlify}[1]{%
  \gdef\hyxmp@xmlified{}%
  % Escaped PDF string -> PDFDocEncoding/Unicode
  \EdefUnescapeString\hyxmp@text{#1}%
  \ifhyxmp@unicodetex
    % PDFDocEncoding/Unicode -> UTF-32BE
    \hyxmp@is@unicode\hyxmp@text{%
      \StringEncodingConvert
      \hyxmp@text\hyxmp@text{utf16be}{utf32be}%
    }{%
      \ifxetex
        \hyxmp@xetex@crap
      \else
        \StringEncodingConvert
        \hyxmp@text\hyxmp@text{pdfdoc}{utf32be}%
      \fi
    }%
    % UTF-32BE -> UTF-32BE as hex string
    \EdefEscapeHex\hyxmp@text{\hyxmp@text}%
    % UTF-32BE -> XML in ASCII
    \edef\hyxmp@text{%
      \expandafter
    }\expandafter\hyxmp@toxml@unicodetex\hyxmp@text
    \relax\relax\relax\relax\relax\relax\relax\relax
  \else
    % PDFDocEncoding/Unicode -> UTF-8
    \hyxmp@is@unicode\hyxmp@text{%
      \StringEncodingConvert
      \hyxmp@text\hyxmp@text{utf16be}{utf8}%
    }{%
      \StringEncodingConvert
      \hyxmp@text\hyxmp@text{pdfdoc}{utf8}%
    }%
    % UTF-8 -> UTF-8 as hex string
    \EdefEscapeHex\hyxmp@text{\hyxmp@text}%
    % UTF-8 as hex string -> XML in UTF-8 as hex string
    \edef\hyxmp@text{%
      \expandafter\hyxmp@toxml\hyxmp@text\@empty\@empty
    }%
    % XML in UTF-8 as hexstring -> XML in UTF-8
    \EdefUnescapeHex\hyxmp@text{\hyxmp@text}%
  \fi
  \global\let\hyxmp@xmlified\hyxmp@text
}
\begingroup
  \lccode`\<=254 %
  \lccode`\>=255 %
  \catcode254=12 %
  \catcode255=12 %
\lowercase{\endgroup
  \def\hyxmp@is@unicode#1{%
    \expandafter\hyxmp@@is@unicode#1<>\@nil
  }%
  \def\hyxmp@@is@unicode#1<>#2\@nil{%
    \ifx\\#1\\%
      \expandafter\@firstoftwo
    \else
      \expandafter\@secondoftwo
    \fi
  }%
}
\def\hyxmp@toxml@unicodetex#1#2#3#4#5#6#7#8{%
  \ifx#1\relax
  \else
    \ifnum"#1#2#3#4#5#6#7#8>127 %
      \uccode`\*="#1#2#3#4#5#6#7#8\relax
      \uppercase{%
        \edef\hyxmp@text{\hyxmp@text *}%
      }%
    \else\ifnum"#7#8=`\< %
      \edef\hyxmp@text{\hyxmp@text &lt;}%
    \else\ifnum"#7#8=`\& %
      \edef\hyxmp@text{\hyxmp@text &amp;}%
    \else\ifnum"#7#8=`\> %
      \edef\hyxmp@text{\hyxmp@text &gt;}%
    \else\ifnum"#7#8=`\ %
      \edef\hyxmp@text{\hyxmp@text\space}%
    \else
      \uccode`\*="#7#8\relax
      \uppercase{%
        \edef\hyxmp@text{\hyxmp@text *}%
      }%
    \fi\fi\fi\fi\fi
    \expandafter\hyxmp@toxml@unicodetex
  \fi
}
\def\hyxmp@skipzeros#1{%
  \ifx#10%
    \expandafter\hyxmp@skipzeros
  \fi
}
\def\hyxmp@toxml#1#2{%
  \ifx#1\@empty
  \else
    \ifnum"#1#2=`\& %
      26616D703B% &amp;
    \else\ifnum"#1#2=`\< %
      266C743B% &lt;
    \else\ifnum"#1#2=`\> %
      2667743B%
    \else
      #1#2%
    \fi\fi\fi
    \expandafter\hyxmp@toxml
  \fi
}

% In case of xetex the strings by \pdfstringdef can contain
% big chars, then the string ist treated as Unicode.
\begingroup
\def\x#1{\endgroup
  \def\hyxmp@xetex@crap{%
    \edef\hyxmp@try{%
      \expandafter\hyxmp@SpaceOther\hyxmp@text#1\@nil
    }%
    \let\hyxmp@crap@result=N%
    \expandafter\hyxmp@crap@test\hyxmp@try\relax
    \ifx\hyxmp@crap@result Y%
      \let\hyxmp@text\@empty
      \expandafter\hyxmp@crap@convert\hyxmp@try\relax
    \else
      \StringEncodingConvert
      \hyxmp@text\hyxmp@text{pdfdoc}{utf32be}%
    \fi
  }%
}\x{ }
\begingroup
  \catcode`\~=12 %
  \lccode`\~=`\ %
\lowercase{\endgroup
  \def\hyxmp@SpaceOther#1 #2\@nil{%
    #1%
    \ifx\relax#2\relax
      \expandafter\@gobble
    \else
      ~%
      \expandafter\@firstofone
    \fi
    {\hyxmp@SpaceOther#2\@nil}%
  }%
}
\def\hyxmp@crap@test#1{%
  \ifx#1\relax
  \else
    \ifnum`#1>127 %
      \let\hyxmp@crap@result=Y%
      \expandafter\expandafter\expandafter\hyxmp@skiptorelax
    \else
      \expandafter\expandafter\expandafter\hyxmp@crap@test
    \fi
  \fi
}
\def\hyxmp@skiptorelax#1\relax{}
\def\hyxmp@crap@convert#1{%
  \ifx#1\relax
  \else
    \edef\hyxmp@num{\number`#1}%
    \ifnum\hyxmp@num>"FFFFFF %
      \lccode`\!=\intcalcDiv{\hyxmp@num}{\number"1000000}\relax
      \lowercase{\edef\hyxmp@text{\hyxmp@text!}}%
      \edef\hyxmp@num{\intcalcMod{\hyxmp@num}{\number"1000000}}%
    \else
      \edef\hyxmp@text{\hyxmp@text\hyxmp@zero}%
    \fi
    \ifnum\hyxmp@num>"FFFF %
      \lccode`\!=\intcalcDiv{\hyxmp@num}{\number"10000}\relax
      \lowercase{\edef\hyxmp@text{\hyxmp@text!}}%
      \edef\hyxmp@num{\intcalcMod{\hyxmp@num}{\number"10000}}%
    \else
      \edef\hyxmp@text{\hyxmp@text\hyxmp@zero}%
    \fi
    \ifnum\hyxmp@num>"FF %
      \lccode`\!=\intcalcDiv{\hyxmp@num}{\number"100}\relax
      \lowercase{\edef\hyxmp@text{\hyxmp@text!}}%
      \edef\hyxmp@num{\intcalcMod{\hyxmp@num}{\number"100}}%
    \else
      \edef\hyxmp@text{\hyxmp@text\hyxmp@zero}%
    \fi
    \ifnum\hyxmp@num>0 %
      \lccode`\!=\hyxmp@num\relax
      \lowercase{\edef\hyxmp@text{\hyxmp@text!}}%
    \else
      \edef\hyxmp@text{\hyxmp@text\hyxmp@zero}%
    \fi
    \expandafter\hyxmp@crap@convert
  \fi
}
\begingroup
  \catcode0=12 %
  \gdef\hyxmp@zero{^^00}%
\endgroup

\def\hyxmp@text{aö€x}
\tracingmacros=1
\hyxmp@xetex@crap
\tracingmacros=0

\renewcommand*{\hyxmp@list@to@xml}[3]{%
  \ifx#3\@empty
  \else
    \hyxmp@add@to@xml{%
_________<dc:#1>^^J%
____________<rdf:#2>^^J%
    }%
    \bgroup
      \hyxmp@xmlify{#3}%
      \hyxmp@commas@to@list\hyxmp@list{\hyxmp@xmlified}%
      \def\@elt##1{%
        \hyxmp@add@to@xml{%
_______________<rdf:li>##1</rdf:li>^^J%
        }%
      }%
      \hyxmp@list
    \egroup
    \hyxmp@add@to@xml{%
____________</rdf:#2>^^J%
_________</dc:#1>^^J%
    }%
  \fi
}

\begingroup
  \ifhyxmp@unicodetex
    \lccode`\!="FEFF %
    \lowercase{%
      \gdef\hyxmp@bom{!}
    }%
  \else
    \catcode`\^^ef=12
    \catcode`\^^bb=12
    \catcode`\^^bf=12
    \gdef\hyxmp@bom{^^ef^^bb^^bf}%
  \fi
\endgroup

\def\hyxmp@construct@packet{%
  \gdef\hyxmp@xml{}%
  \hyxmp@add@to@xml{<?xpacket begin="\hyxmp@bom" %
id="W5M0MpCehiHzreSzNTczkc9d"?>^^J%
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="3.1-702">^^J%
___<rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns\hyxmp@hash">^^J%
  }%
  \hyxmp@pdf@schema
  \hyxmp@xmpRights@schema
  \hyxmp@dc@schema
  \hyxmp@photoshop@schema
  \hyxmp@mm@schema
  \hyxmp@add@to@xml{%
___</rdf:RDF>^^J%
</x:xmpmeta>^^J%
\hyxmp@padding
<?xpacket end="w"?>^^J%
  }%
}

\renewcommand*{\hyxmp@reencode}[1]{%
  % dummy only used for \@pdfmetalang in \begin{document}
}
\AtBeginDocument{%
  \@ifpackageloaded{hyperref}{%
    \ifx\@pdflang\@empty
      \let\@pdfmetalang=\hyxmp@x@default
    \else
      \edef\@pdfmetalang{\@pdflang}%
    \fi
    \hyxmp@xmlify\@pdfmetalang
  }{}%
}
\makeatother

\usepackage{lipsum}

\begin{document}
\lipsum[1-3]
\end{document}

I have not tested all possible cases and drivers, thus there might be some unrevealed problems. But maybe the patch helps in improving the package.

Answer 2

This is solved as of 2020. You also do not need to specify the driver of hyperref manually any more.