19

As described in The TeXbook, TeX reads files byte by byte, regardless of the particular format -- as I understand, this is just how IniTeX is set up.

I also understand that LaTeX is just a collection of macros built on top of IniTeX, described in most distributions of TeX by the file latex.ltx.

The above two things are at odds with my understanding of LaTeX's ability to read UTF-8. I was under the impression that reading the input byte by byte (and thus for instance, only being able to access numbers from 0 to 255 using \char or something) was baked into TeX, and thus would exist in all variants built on top of it.

Thus, how is LaTeX able to do this?

  • 5
    The TeXbook describes the so-called Knuth-TeX engine, as well as a collection of macros frequently called "PlainTeX". Are you aware of newer engines called pdfTeX, XeTeX, and LuaTeX? – Mico Jan 21 at 1:54
  • 3
    You could be asking one of (at least) two questions here. Are you wondering how e.g. XeTeX (natively UTF-8) can be derived from Knuth's TeX (8-bit). Or are you wondering how 8-bit TeX engines deal with UTF-8 input (conversion of 'raw' bytes to codpoints to output)? – Joseph Wright Jan 21 at 7:42
  • @JosephWright The former. – extremeaxe5 Jan 22 at 18:40
  • @Mico I am aware they exist. Are you saying that the newer TeX engines read input utf8 char by utf8 char? How would this work? I feel like a lot of things would break -- as Knuth describes, catcodes, mathcodes, and delcodes are only defined for 256 distinct values (all the possible bytes), no? – extremeaxe5 Jan 22 at 18:43
  • Related: tex.stackexchange.com/questions/222286/…. There, I look in detail at the differences between engines, some of which are linked to Unicode support. (The questions are distinct, it's not a duplicate.) – Joseph Wright Jan 22 at 20:52
20

If you want to know how the 8-bit engines handle utf8 input you can use \tracingmacros:

\documentclass{article}

\begin{document}
{\tracingmacros =1 ä }
\end{document}

which gives

Ã->\UTFviii@two@octets Ã

\UTFviii@two@octets #1#2->\expandafter \UTFviii@defined \csname u8:#1\string #2
\endcsname 
#1<-Ã
#2<-¤

\UTFviii@defined #1->\ifx #1\relax \if \relax \expandafter \UTFviii@checkseq \s
tring #1\relax \relax \UTFviii@undefined@err {#1}\else \PackageError {inputenc}
{Invalid UTF-8 byte sequence}\UTFviii@invalid@help \fi \else \expandafter #1\fi

#1<-\u8:ä 

\u8:ä ->\IeC {\"a}

That means the the first byte of the ä (the Ã) is an active char, a command which then picks up the next byte and then calls \u8:ä which calls \"a. In this way (pdf)latex can handle quite a lot of utf8 input but it has e.g. problems with "char + combining accent" as there is no sensible code for the combining accent to go back to add an accent on the char.

  • 1
    Just a comment to point out log file containing the trace got viewed by editor in some 8bit encoding, presumably iso-latin-1 (like Emacs does for me), not in UTF8... so à is only one byte. This is tacit in your answer... – user4686 Jan 21 at 9:01
5

In order to move from 8-bit TeX90 to Unicode XeTeX or LuaTeX, there is work to do in extending/modifying internal structures. However, that is largely a question of effort rather than any major conceptual limitation. Knuth after all extended TeX from 7-bit to 8-bit between TeX82 (TeX 2) and TeX90 (TeX 3).

Both XeTeX and LuaTeX read files in UTF-8 rather than on a per-byte basis. This happens well before any TeX-related processes are involved, and as such at the macro level there are only UTF-8 characters. (One can alter this in LuaTeX: see luainputenc for example.) Both engines then use tables which are extended to cover the full Unicode range.

The change in accepted input can be used to test for Unicode-aware engines, as shown in an example from https://www.contextgarden.net/Encodings_and_Regimes

\def\test#1#2!{\def\secondarg{#2}}
\test χ!\relax % That's Chi, a 2-byte utf-8 sequence
\ifx\secondarg\empty \message{newstuff}\else \message{tex82}\fi

In the main, macro code does not need altering to accept Unicode: the engines deal with the byte aspect, so from a macro viewpoint everything is 'as expected'. Of course, there is a little set up to do, for example setting \catcode, \uccode, etc. for the full Unicode range. Today, this is handled by using the unicode-data, so is built-in to both plain TeX-derived and LaTeX formats.

There are a few places that LaTeX has to be aware of which engine is in use, but the direct impact of Unicode is largley limited to

  • Setting up the data for \catcode, etc.
  • Setting up hyphenation patterns (which today are all stored as UTF-8 and require more work to use with pdfTeX than with Unicode TeX engines)
  • (Not) setting up support for Unicode support based on active 8-bit characters (See Ulrike's answer for details of how that works in 8-bit engines)

Other aspects at the macro layer are related to other functionality, for example the ability of both XeTeX and LuaTeX to load system fonts: that requires a Unicode font encoding (TU), but that is distinct from handling input.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.