# Why does xunicode not provide true backward compatibility?

As it is said in xetex mailing list

Xunicode is mostly about backward compatibility, providing sensible general definitions for macros that have been defined in older packages which now should not be used with XeTeX. This allows LaTeX source to be reused, and does not force people to learn new input methods, when they have already built up experience with non-XeTeX versions of LaTeX, etc.

I wonder then why xunicode emulates LaTeX accents by a method which differes from natural LaTeX method (see description here):

xunicode takes characters entered through the traditional TeX keystrokes (\'e for e-acute, etc.) and places the precomposed Unicode character (U+00E9 for e-acute) in the output file. (In other words, it doesn't really matter whether you enter e-acute directly, if you have a keyboard that supports it, or using the TeX keystrokes; you still get the Unicode precomposed character.) If the combination that you type using traditional TeX methods does not exist in Unicode in precomposed form (e.g., \v y, since y-caron is not a precomposed combination), xunicode inserts the combining mark after the base letter. I also notice that if I enter 'e' followed by \char"0301, the combining mark remains (i.e., xunicode does not replace this sequence with the precomposed version.)

As a result, output is not satisfactory for some of those fonts which do not have precomposed accented charcters in case if e-acute is absent.

I mean why xunicode outputs e\char"0301 instead of emulating true LaTeX definition of the \'{e} command?

-
Good question. Personally I think xunicode is not very well implemented. –  Leo Liu Sep 12 '11 at 15:32
Not sure about implementation of xunicode, but it is not even documented. –  Igor Kotelnikov Sep 12 '11 at 15:44
@Leo — we might be thinking of adapting some of xunicode to support font encodings as part of the LaTeX3 project. Do you have any feedback on what could be improved? (Email probably better.) –  Will Robertson Sep 13 '11 at 5:01

I'm not sure if anyone can answer this in a concrete sense except Ross Moore. My impression is that you'll get better output with most modern fonts using the combining glyph. If you're not I'd consider that a font bug.

Regarding:

I also notice that if I enter e followed by \char"0301, the combining mark remains (i.e., xunicode does not replace this sequence with the precomposed version.)

xunicode cannot reasonably do anything in this case — it only has control over the accent macros themselves.

-
I may have been mistaken about the behaviour of combining glyphs; see Igor's answer for another take on the matter! –  Will Robertson Sep 13 '11 at 5:00

I would not agree with Will Robertson ragarding his statement that most modern fonts give better output for method used in xunicode by default in case when precomposed accented symbol is absent from the font [sorry, I don't know how to call that method; I mean that typing e\char"0301 in source file outputs accented e].

1. In my Windows 7 this default method produces correct placement of accents only for those fonts that were absent in previous versions of Windows (in particular: Cambria, Constantia, Corbel). Even Times New Roman and Palatino Linotype produce misplaced accents over cyrillic letters. Nothing to say about fonts supplied not by Microsoft. Adobe Arno Pro is very recent font but it has same troubles.

2. More over, this default method outputs nothing (or crossed out box or question mark) for many accent commands / font combinations (\c, \b, \r, \d, \v, \=).

Both these troubles can be solved for most fonts I tested so far by adaptating standard LaTeX's definitions of the accent commands for EU1 encoding. Here is my code:

\DeclareTextAccent{\}{EU1}{"0300}
\DeclareTextAccent{\'}{EU1}{"0301}
\DeclareTextAccent{\^}{EU1}{"02C6}
\DeclareTextAccent{\~}{EU1}{"02DC}
\DeclareTextAccent{\=}{EU1}{"02C9}
\DeclareTextAccent{\u}{EU1}{"02D8}
\DeclareTextAccent{\.}{EU1}{"02D9}
\DeclareTextAccent{\"}{EU1}{"00A8}
\DeclareTextAccent{\r}{EU1}{"02DA}
\DeclareTextAccent{\H}{EU1}{"02DD}
\DeclareTextAccent{\v}{EU1}{"02C7}
\makeatletter
\DeclareTextCommand{\b}{EU1}[1]
{\hmode@bgroup\o@lign{\relax#1\crcr\hidewidth\ltx@sh@ft{-3ex}%
\vbox to.2ex{\hbox{\char"005F}\vss}\hidewidth}\egroup}
\DeclareTextCommand{\c}{EU1}[1]
{\leavevmode\setbox\z@\hbox{#1}\ifdim\ht\z@=1ex\accent11 #1%
\else{\ooalign{\unhbox\z@\crcr
\hidewidth\char"00B8\hidewidth}}\fi}
\DeclareTextCommand{\d}{EU1}[1]
{\hmode@bgroup
\o@lign{\relax#1\crcr\hidewidth\ltx@sh@ft{-1ex}.\hidewidth}\egroup}
\DeclareTextCommand{\k}{EU1}[1]
{\hmode@bgroup\ooalign{\null#1\crcr\hidewidth\char"02DB}\egroup}
\DeclareTextCommand{\textogonekcentered}{EU1}[1]
{\hmode@bgroup\ooalign{\null#1\crcr\hidewidth\char"02DB\hidewidth}\egroup}
\makeatother


Again, Arno Pro is out of common range:  and \' as defined above fail for this font since  and ' are in slots 0060 and 00B4 respectively.

-
You might be right! I simply assumed most fonts did sensible things with combining accents since I'd never heard otherwise. IMO this should be a choice of the user which accent method to use — we could have, say, EU3 (or EU0) for a font encoding with ‘fake’ accents. I can't do it myself, but if the package is written I'll support it in fontspec! –  Will Robertson Sep 13 '11 at 4:59
@Will: what else should I do to make this code reusable? Wrap it into a separate package? If so, could you propose a name for this package. IMO this package could be loaded by fontspec if fontspec is requested with an option. Then the name of this option should somehow be related to the name of package. –  Igor Kotelnikov Sep 13 '11 at 15:37
Igor — good question! I suppose the most sensible thing to do would be to add a third encoding to the euenc package, incorporating these definitions in a file similar to t1enc.def. Contact me for write access to github.com/wspr/euenc if you're interested. (Would you be willing to extend the definitions to cover other accents? I assume there are more.) –  Will Robertson Sep 15 '11 at 6:46
@Will, I am wondering why it should be one more encoding? Why it couldn't be an option for xunicode package? Perhaps, it is better to proceed this discussion in a more specialised forum. If github.com allows such discussions then please give me access. You have my email. –  Igor Kotelnikov Sep 16 '11 at 15:13

A simple but real problem is, for example, not every font has accented alphabets. Suppose we want to typeset the name of author of pdfTeX, Hàn Thế Thành: very few fonts have symbol ế (U+1EBF), and it is difficult to type on most keyboards. More over, xunicode cannot handle double accents well. I want to use \'{\^e} or \'{ê}, but both failed, even there is ế glyph in the font.

Test code:

\documentclass{article}
\usepackage{fontspec}
\begin{document}
Latin Modern: {\LARGE \'{\^e}, \'{ê}, ế}

\fontspec{XITS}
XITS:  {\LARGE \'{\^e}, \'{ê}, (ế)}

\fontspec{Minion Pro}
Minion Pro:  {\LARGE \'{\^e}, \'{ê}, ế}
\end{document}


So what can xunicode be? I think there can be a test using \iffontchar. Then xunicode` can decide which method can be used: a single glyph or composed glyphs.

-
detecting combining glyphs using \iffontchar ... requires that every character of the source be thus scanned and converted in case of a combining char; which is all very well, but may involve yet more contortion in an already complicated package. ross could say for sure, but my impression is that would be pretty painful. quite how one might deal with these things is another matter (the double accent in the example is surely spurious, since there's an established technique to get the combined accents right in vanilla tex (it's needed for thanh's name), and the output can be made acceptable –  wasteofspace Sep 13 '11 at 10:06