Is XeTeX a complete rewrite of LaTeX/TeX?

Question

I have been looking into XeTeX for type-setting Unicode Text. I looked into the source code of it. XeTeX source distribution comes with Web2C. Is XeTeX a complete rewrite of TeX? If not, how did it accomplish the goal of adding the support for Unicode?

Answer 1

No, XeTeX was not created by completely rewriting TeX. Rather, it was created by starting from e-TeX and making changes using a .ch file (WEB change file). The change file tackles two big areas: extending the internals of (e-)TeX for Unicode, and extending font loading. The current TeX Live XeTeX source features both this .ch file and an equivalent .web file.

Answer 2

XeTeX is not a complete rewriting of TeX. Here are some of the main changes.

1. The input stage. XeTeX by default reads Unicode files (UTF-8, for instance), although it's capable of interpreting also differently encoded files (for backwards compatibility). Multibyte characters are reduced to a single internal character upon reading, so they are considered as a unique entity when tokenization is performed; you can have command names in cyrillic, if you want, but I wouldn't recommend this practice.

2. The font management. The font management part has been completely reworked by adding support for OpenType and TrueType fonts, delegating some parts to the fontconfig library (and to the Apple font library in case XeTeX is run on Mac OS X).

3. The math font set up. XeTeX introduces new primitives for extending the \mathcode and \mathchardef features in TeX and allowing for specifying characters in the whole Unicode set and in 256 math families (instead of TeX's 16).

4. "Post-processing" features (A). XeTeX links to the teckit library so it can apply a .map file that allows for transforming characters in already formed token lists before they are processed in the "stomach" for typesetting. This is how Ligatures=TeX works, attaching to the font a map directive that transforms the usual combinations into a single character, for instance --- into —.

5. "Post-processing" features (B). Characters can be assigned to an "interchar token class" and it's possible to specify tokens to be added when there is a transition from a class to another. The packages polyglossia, xeCJK and ucharclasses exploit this feature.

The process of typesetting, instead, is essentially the same as TeX's. However some changes have been made also in the hyphenation stage that may give slightly different results if the same file is compiled with pdftex or xetex.

Answer 3

XeTeX is a full utf8 Version of TeX and available with all TeX distributions. The only difference is the font handling, because XeTeX can handle OpenType as well as TrueType fonts:

\documentclass{article}
\usepackage{fontspec}
\setmainfont{DejaVu Serif}
\newfontface\Chinese{KaiTi_GB2312}% simkai.ttf

\begin{document}
Это мой первый многоязычный доклад.

This is my first multilingual report.

\Chinese
这是我的第一个多语种的报告。

\end{document}