How can (La)TeX read UTF-8?

Question

As described in The TeXbook, TeX reads files byte by byte, regardless of the particular format -- as I understand, this is just how IniTeX is set up.

I also understand that LaTeX is just a collection of macros built on top of IniTeX, described in most distributions of TeX by the file latex.ltx.

The above two things are at odds with my understanding of LaTeX's ability to read UTF-8. I was under the impression that reading the input byte by byte (and thus for instance, only being able to access numbers from 0 to 255 using \char or something) was baked into TeX, and thus would exist in all variants built on top of it.

Thus, how is LaTeX able to do this?

Answer 1

If you want to know how the 8-bit engines handle utf8 input you can use \tracingmacros:

\documentclass{article}

\begin{document}
{\tracingmacros =1 ä }
\end{document}

which gives

Ã->\UTFviii@two@octets Ã

\UTFviii@two@octets #1#2->\expandafter \UTFviii@defined \csname u8:#1\string #2
\endcsname 
#1<-Ã
#2<-¤

\UTFviii@defined #1->\ifx #1\relax \if \relax \expandafter \UTFviii@checkseq \s
tring #1\relax \relax \UTFviii@undefined@err {#1}\else \PackageError {inputenc}
{Invalid UTF-8 byte sequence}\UTFviii@invalid@help \fi \else \expandafter #1\fi

#1<-\u8:ä 

\u8:ä ->\IeC {\"a}

That means the the first byte of the ä (the Ã) is an active char, a command which then picks up the next byte and then calls \u8:ä which calls \"a. In this way (pdf)latex can handle quite a lot of utf8 input but it has e.g. problems with "char + combining accent" as there is no sensible code for the combining accent to go back to add an accent on the char.

Answer 2

In order to move from 8-bit TeX90 to Unicode XeTeX or LuaTeX, there is work to do in extending/modifying internal structures. However, that is largely a question of effort rather than any major conceptual limitation. Knuth after all extended TeX from 7-bit to 8-bit between TeX82 (TeX 2) and TeX90 (TeX 3).

Both XeTeX and LuaTeX read files in UTF-8 rather than on a per-byte basis. This happens well before any TeX-related processes are involved, and as such at the macro level there are only UTF-8 characters. (One can alter this in LuaTeX: see luainputenc for example.) Both engines then use tables which are extended to cover the full Unicode range.

The change in accepted input can be used to test for Unicode-aware engines, as shown in an example from https://www.contextgarden.net/Encodings_and_Regimes

\def\test#1#2!{\def\secondarg{#2}}
\test χ!\relax % That's Chi, a 2-byte utf-8 sequence
\ifx\secondarg\empty \message{newstuff}\else \message{tex82}\fi

In the main, macro code does not need altering to accept Unicode: the engines deal with the byte aspect, so from a macro viewpoint everything is 'as expected'. Of course, there is a little set up to do, for example setting \catcode, \uccode, etc. for the full Unicode range. Today, this is handled by using the unicode-data, so is built-in to both plain TeX-derived and LaTeX formats.

There are a few places that LaTeX has to be aware of which engine is in use, but the direct impact of Unicode is largley limited to

• Setting up the data for \catcode, etc.
• Setting up hyphenation patterns (which today are all stored as UTF-8 and require more work to use with pdfTeX than with Unicode TeX engines)
• (Not) setting up support for Unicode support based on active 8-bit characters (See Ulrike's answer for details of how that works in 8-bit engines)

Other aspects at the macro layer are related to other functionality, for example the ability of both XeTeX and LuaTeX to load system fonts: that requires a Unicode font encoding (TU), but that is distinct from handling input.