# Is XeTeX a complete rewrite of LaTeX/TeX?

I have been looking into XeTeX for type-setting Unicode Text. I looked into the source code of it. XeTeX source distribution comes with Web2C. Is XeTeX a complete rewrite of TeX? If not, how did it accomplish the goal of adding the support for Unicode?

-

No, XeTeX was not created by completely rewriting TeX. Rather, it was created by starting from e-TeX and making changes using a .ch file (WEB change file). The change file tackles two big areas: extending the internals of (e-)TeX for Unicode, and extending font loading. The current TeX Live XeTeX source features both this .ch file and an equivalent .web file.

-
I'm not sure quite how everything fits together for the file reading stuff: I have enough trouble even reading WEB files, let alone linking together with other stuff :-) –  Joseph Wright Nov 19 '12 at 13:42
Actually in SVN the main change file is already merged so xetex.web is now the canonical source (this wasn’t the case few weeks ago, we didn’t even have a xetex.web files it was created at build time), the remaining change file is for some minor stuff. –  Khaled Hosny Nov 19 '12 at 17:25

XeTeX is not a complete rewriting of TeX. Here are some of the main changes.

1. The input stage. XeTeX by default reads Unicode files (UTF-8, for instance), although it's capable of interpreting also differently encoded files (for backwards compatibility). Multibyte characters are reduced to a single internal character upon reading, so they are considered as a unique entity when tokenization is performed; you can have command names in cyrillic, if you want, but I wouldn't recommend this practice.

2. The font management. The font management part has been completely reworked by adding support for OpenType and TrueType fonts, delegating some parts to the fontconfig library (and to the Apple font library in case XeTeX is run on Mac OS X).

3. The math font set up. XeTeX introduces new primitives for extending the \mathcode and \mathchardef features in TeX and allowing for specifying characters in the whole Unicode set and in 256 math families (instead of TeX's 16).

4. "Post-processing" features (A). XeTeX links to the teckit library so it can apply a .map file that allows for transforming characters in already formed token lists before they are processed in the "stomach" for typesetting. This is how Ligatures=TeX works, attaching to the font a map directive that transforms the usual combinations into a single character, for instance --- into —.

5. "Post-processing" features (B). Characters can be assigned to an "interchar token class" and it's possible to specify tokens to be added when there is a transition from a class to another. The packages polyglossia, xeCJK and ucharclasses exploit this feature.

The process of typesetting, instead, is essentially the same as TeX's. However some changes have been made also in the hyphenation stage that may give slightly different results if the same file is compiled with pdftex or xetex.

-
So font management is partly delegated to OS-specific libraries (fontconfig vs. Apple font library). Does this imply any danger that the typesetting result may be different on MacOS than on other systems? (I remember horrible experiences with OpenOffice Impress on Linux, Windows and MacOS in this respect, which triggered my conversion to LaTeX beamer) –  Daniel Jan 30 '13 at 16:10
@Daniel See this recent thread about the problem –  egreg Jan 30 '13 at 16:14
Thanks, exactly what I was looking for. Don't know how I could miss it. –  Daniel Jan 30 '13 at 19:58

XeTeX is a full utf8 Version of TeX and available with all TeX distributions. The only difference is the font handling, because XeTeX can handle OpenType as well as TrueType fonts:

\documentclass{article}
\usepackage{fontspec}
\setmainfont{DejaVu Serif}
\newfontface\Chinese{KaiTi_GB2312}% simkai.ttf

\begin{document}
Это мой первый многоязычный доклад.

This is my first multilingual report.

\Chinese

\end{document}


-
What is the difference at the source code level? –  codefx Nov 19 '12 at 10:08
The font handling ... –  Herbert Nov 19 '12 at 10:12
So it uses exactly the TeX's expansion, glue-handling and breaking-handling algorithms? –  yo' Nov 19 '12 at 10:23
@tohecz: as far as OpenType features are not involved you'll get the same output –  Herbert Nov 19 '12 at 10:37
@Herbert your several rollbacks all break formatting… let the Chinese text be inside the block quote. –  F'x Nov 19 '12 at 14:37