In The LaTeX Companion (Addison-Wesley 2004) Mittlebach & Gossens' dedicated section 7.11.4 to discussing output encodings: the mapping from LICR, the internal representation of characters and symbols inside a LaTeX engine, to glyphs or combinations of glyphs available in a font file.

Since publication of the book a new generation of LaTeX engines, xetex and luatex, have come to the foreground. Unlike the old engines, the new ones make it easy - thanks to the fontspec package - to use any font installed on one's computer. Moreover, unlike the fontenc package, which was described in The LaTeX Companion as the cornerstone of the output encoding mechanism, fontspec requires no encoding (e.g. OT1, T1) to be specified.

While the new engines, unlike the traditional ones, read UTF-8-encoded input files, which makes specifying an input encoding unnecessary, the internal representation of the input inside the engines hasn't changed, as far as I know, yet the number of available fonts has increased dramatically. So it would seem that the problem of specifying an output encoding would remain as relevant to the new engines as it was (and is) to the traditional ones.

How does the process of output encoding, of mapping LICR to glyphs, work in the new engines?

  • 3
    On both side it is "unicode". Jun 25 '17 at 19:47
  • 1
    @UlrikeFischer: Except when it's not.
    – Evan Aad
    Jun 25 '17 at 20:36
  • 1
    If you have a special case in mind you should specify your question. Jun 25 '17 at 20:45
  • 1
    Note that it is not an engine thing per se. You will get a different result for pdfTeX if you load a different encoding e.g. \usepackage[T1]{fontenc}. It is just that the default encoding is engine-dependent. So pdfTeX gives you OT1, whereas XeTeX gives you TU.
    – cfr
    Jun 25 '17 at 21:38
  • 1
    If you want to use a type1 font (which doesn't know unicode) you should use the standard legacy method to access it (through tfm files and map-files). \setmainfont doesn't make much sense, as you can access the glyph only by slots and this is very font dependant (in your example \XeTeXglyph 2 would give you the lambda.) Jun 25 '17 at 21:53

In many ways, font encodings in XeTeX and LuaTeX work the same as in pdfTeX, etc. One loads a font and has to know how the glyphs are encoded so that the input can be mapped to the correct output.

The big difference is that both XeTeX and LuaTeX can load OpenType system fonts (.otf files). Unlike 'traditional' TeX fonts, which come in a range of encodings and have at most 256 slots available per font, OpenType fonts are laid out in Unicode. As both of these engines also use Unicode as their standard input encoding, this means there is a direct mapping from input to output and no manipulation is needed.

If one wishes to use a non-Unicode font with XeTeX or LuaTeX, the same approaches as for pdfTeX are required: pick the correct encoding and set up appropriate mechanisms. For LaTeX, that would mean using fontenc. Certainly it is possible to load a 'classical' .tfm font with XeTeX or LuaTeX and to get the 'expected' glyphs out.

However, there are some caveats. The key one is that hyphenation patterns are based on the font encoding, and for 'classical' TeX engine including XeTeX can only be read when making a format. This means that when making a format one needs to know something about the font encodings that will be used. As XeTeX is a Unicode engine, it loads the hyphenation patterns in a Unicode encoding. This requires that the fonts used are also Unicode-encoded if hyphenation is to be correct for all languages. The number of codepoints this affects is small (largely Unicode and T1 overlap), but it is an issue. For that reason, the LaTeX team relativity recently changed the default font encoding in XeTeX (and LuaTeX) from OT1 to Unicode: this may catch out the unwary trying low-level font manipulation.

It is worth noting that LuaTeX is a slightly different case to XeTeX as it can load hyphenation patterns at run time, and as it allows re-encoding of the input. As such, LuaTeX is in some ways more able to cope with non-Unicode input than XeTeX is. However, LuaTeX development is also very much focussed on a purely-Unicode pathway, and I would be wary of creating any new documents heavily using classical TeX fonts with LuaTeX as a result.


Here's what I take from the discussion in the comments (thanks to Ulrike Fischer, cfr and Joseph Wright, though any misunderstanding/misrepresentation is mine).

The fontspec package is intended to be used with unicode fonts. For other fonts use fontenc instead, just as you would with pdflatex.

How to tell that a font is unicode? If it's Open Type, it's unicode. The file extensions .otf and .otc imply an OpenType font. The .ttf and .ttc extensions can likewise mean an OpenType font, however they can also mean a regular (non-OpenType) TrueType font, so caveat emptor.

  • 2
    Taking information from comments and posting it as for credit yourself is rather rude. If the comments answered you, ask somebody to write an answer. The font's font encoding isn't relevant. That's what I was trying to say. For traditional fonts, it may be relevant, if it happens that the font is designed specifically for TeX and doesn't need reencoding. But most type1 fonts are not in this category. They are reencoded. So a typical type1 font is not intrinsically OT1 or T1. It may be Adobe, for example (in the most standard case). But you don't use the Adobe encoding to access the font.
    – cfr
    Jun 25 '17 at 23:42
  • it might be worth remembering that tex predates unicode, and since many tex users have legacy documents that they wish to continue to be usable, the original font mechanism must remain available. Jun 25 '17 at 23:43
  • Look at most type1 fonts and you will see that the encoding is not something you'd use in TeX. yvtr8a.afm: 8a for the Adobe encoding. The file includes EncodingScheme AdobeStandardEncoding. You don't use the AFM directly, of course, but it is used to generate the files you do. But you can't use this encoding directly in TeX. You might use OT1 or T1 or whatever. Typically, you may use different output encodings with the same postscript type1 font. The font has an encoding, but it is irrelevant unless you are creating the support package.
    – cfr
    Jun 25 '17 at 23:46
  • @barbarabeeton And some of us don't trust the new-fangled fonts!
    – cfr
    Jun 25 '17 at 23:48
  • 1
    @cfr: Go ahead and write an answer. I'll accept it and delete mine.
    – Evan Aad
    Jun 26 '17 at 4:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.