Why does xunicode not provide true backward compatibility?

Question

As it is said in xetex mailing list

Xunicode is mostly about backward compatibility, providing sensible general definitions for macros that have been defined in older packages which now should not be used with XeTeX. This allows LaTeX source to be reused, and does not force people to learn new input methods, when they have already built up experience with non-XeTeX versions of LaTeX, etc.

I wonder then why xunicode emulates LaTeX accents by a method which differes from natural LaTeX method (see description here):

xunicode takes characters entered through the traditional TeX keystrokes (\'e for e-acute, etc.) and places the precomposed Unicode character (U+00E9 for e-acute) in the output file. (In other words, it doesn't really matter whether you enter e-acute directly, if you have a keyboard that supports it, or using the TeX keystrokes; you still get the Unicode precomposed character.) If the combination that you type using traditional TeX methods does not exist in Unicode in precomposed form (e.g., \v y, since y-caron is not a precomposed combination), xunicode inserts the combining mark after the base letter. I also notice that if I enter 'e' followed by \char"0301, the combining mark remains (i.e., xunicode does not replace this sequence with the precomposed version.)

As a result, output is not satisfactory for some of those fonts which do not have precomposed accented charcters in case if e-acute is absent.

I mean why xunicode outputs e\char"0301 instead of emulating true LaTeX definition of the \'{e} command?

Answer 1

I'm not sure if anyone can answer this in a concrete sense except Ross Moore. My impression is that you'll get better output with most modern fonts using the combining glyph. If you're not I'd consider that a font bug.

Regarding:

I also notice that if I enter e followed by \char"0301, the combining mark remains (i.e., xunicode does not replace this sequence with the precomposed version.)

xunicode cannot reasonably do anything in this case — it only has control over the accent macros themselves.

Answer 2

I would not agree with Will Robertson ragarding his statement that most modern fonts give better output for method used in xunicode by default in case when precomposed accented symbol is absent from the font [sorry, I don't know how to call that method; I mean that typing e\char"0301 in source file outputs accented e].

1. In my Windows 7 this default method produces correct placement of accents only for those fonts that were absent in previous versions of Windows (in particular: Cambria, Constantia, Corbel). Even Times New Roman and Palatino Linotype produce misplaced accents over cyrillic letters. Nothing to say about fonts supplied not by Microsoft. Adobe Arno Pro is very recent font but it has same troubles.

2. More over, this default method outputs nothing (or crossed out box or question mark) for many accent commands / font combinations (\c, \b, \r, \d, \v, \=).

Both these troubles can be solved for most fonts I tested so far by adaptating standard LaTeX's definitions of the accent commands for EU1 encoding. Here is my code:

\DeclareTextAccent{\`}{EU1}{"0300}
\DeclareTextAccent{\'}{EU1}{"0301}
\DeclareTextAccent{\^}{EU1}{"02C6}
\DeclareTextAccent{\~}{EU1}{"02DC}
\DeclareTextAccent{\=}{EU1}{"02C9}
\DeclareTextAccent{\u}{EU1}{"02D8}
\DeclareTextAccent{\.}{EU1}{"02D9}
\DeclareTextAccent{\"}{EU1}{"00A8}
\DeclareTextAccent{\r}{EU1}{"02DA}
\DeclareTextAccent{\H}{EU1}{"02DD}
\DeclareTextAccent{\v}{EU1}{"02C7}
\makeatletter
\DeclareTextCommand{\b}{EU1}[1]
   {\hmode@bgroup\o@lign{\relax#1\crcr\hidewidth\ltx@sh@ft{-3ex}%
     \vbox to.2ex{\hbox{\char"005F}\vss}\hidewidth}\egroup}
\DeclareTextCommand{\c}{EU1}[1]
   {\leavevmode\setbox\z@\hbox{#1}\ifdim\ht\z@=1ex\accent11 #1%
     \else{\ooalign{\unhbox\z@\crcr
        \hidewidth\char"00B8\hidewidth}}\fi}
\DeclareTextCommand{\d}{EU1}[1]
   {\hmode@bgroup
    \o@lign{\relax#1\crcr\hidewidth\ltx@sh@ft{-1ex}.\hidewidth}\egroup}
\DeclareTextCommand{\k}{EU1}[1]
   {\hmode@bgroup\ooalign{\null#1\crcr\hidewidth\char"02DB}\egroup}
\DeclareTextCommand{\textogonekcentered}{EU1}[1]
   {\hmode@bgroup\ooalign{\null#1\crcr\hidewidth\char"02DB\hidewidth}\egroup}
\makeatother

Again, Arno Pro is out of common range: ` and \' as defined above fail for this font since ` and ' are in slots 0060 and 00B4 respectively.

Answer 3

A simple but real problem is, for example, not every font has accented alphabets. Suppose we want to typeset the name of author of pdfTeX, Hàn Thế Thành: very few fonts have symbol ế (U+1EBF), and it is difficult to type on most keyboards. More over, xunicode cannot handle double accents well. I want to use \'{\^e} or \'{ê}, but both failed, even there is ế glyph in the font.

Test code:

\documentclass{article}
\usepackage{fontspec}
\begin{document}
Latin Modern: {\LARGE \'{\^e}, \'{ê}, ế}

\fontspec{XITS}
XITS:  {\LARGE \'{\^e}, \'{ê}, (ế)}

\fontspec{Minion Pro}
Minion Pro:  {\LARGE \'{\^e}, \'{ê}, ế}
\end{document}

So what can xunicode be? I think there can be a test using \iffontchar. Then xunicode can decide which method can be used: a single glyph or composed glyphs.