How is the TeX engine informed of the OS's end-of-line marker and file encoding?

Question

The TeX engine processes the input file on a line-by-line basis (The TEXbook, p. 46). The first line will be read from the file, undergo some pre-processing (see below), and then the engine will start processing it. When the line has been processed, the next line will be read from the file, etc.

The procedure that reads the next line from the file is effectively separate and independent from the TeX engine; in particular, it is "oblivious" to the concept of category codes, and has effectively no access to the catcode table. Therefore, as far as this procedure is concerned, a line of input is defined the way the operating system defines it. For instance, on a Windows system the end of a line is indicated with a CR-LF combination, whereas on a Linux system it is indicated with LF. The end-of-line marker, if any, is not included in the list of bytes returned by the reading procedure (TeX by Topic, p. 29).

After the line is read in, and before the engine begins to process it, the line undergoes some pre-processing:

1. (The TEXbook, p. 43) The characters are converted from whatever encoding they were represented in inside the input file to the ASCII encoding. For instance, if the file was encoded with EBCDIC, then a byte representing the number 129 (which is EBCDIC's code-point for lowercase a), will be replaced by a byte representing the number 97 (which is ASCII's code-point for lowercase a).
2. (The TEXbook, p. 46) Any trailing spaces (ASCII 32) are dropped.
3. (The TEXbook, p. 46) A Carriage Return (ASCII 13) is appended at the end of the line.

The line is now ready to be processed by the TeX engine, and is handed over to the tokenizer.

As the description above shows, the TeX engine depends on the following parameters:

• The OS's end-of-line marker
• The input file's encoding

and cannot even begin to process the input file without knowing their values.

1. Are these parameters customizable by the user, or are they hard-coded in the source code? If it is the former, how can they be customized for, say, pdftex?

2. If it is the latter, in what sense are they hard-coded:

   1. In the sense that every distribution is built from a slightly different source code customized to a certain end-of-line marker/file encoding configuration?
   2. In the sense that the source code tests for these values during run-time?
   3. In the sense that the source code uses some portable API for reading lines and characters?

---

Cited references

• Knuth, Donald Ervin. The TEXbook: Addison-Wesley, 1991.
• Eijkhout, Victor. TeX by Topic: Addison-Wesley, 1992.

Answer 1

The record terminator is OS dependent, but Web2C based TeX distributions (TeX Live and MiKTeX) can decide it on the fly. I created three identical files containing

abc
def
\bye

but with different line terminators; here are the hex dumps:

> hexdump testeol-lf.tex 
0000000 61 62 63 0a 64 65 66 0a 5c 62 79 65 0a         
000000d
> hexdump testeol-cr.tex 
0000000 61 62 63 0d 64 65 66 0d 5c 62 79 65 0d         
000000d
> hexdump testeol-crlf.tex 
0000000 61 62 63 0d 0a 64 65 66 0d 0a 5c 62 79 65 0d 0a
0000010

If I compile them, I get

> pdftex testeol-lf
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=pdftex)
 restricted \write18 enabled.
entering extended mode
(./testeol-lf.tex [1{/usr/local/texlive/2017/texmf-var/fonts/map/pdftex/updmap/
pdftex.map}] )</usr/local/texlive/2017/texmf-dist/fonts/type1/public/amsfonts/c
m/cmr10.pfb>
Output written on testeol-lf.pdf (1 page, 10842 bytes).
Transcript written on testeol-lf.log.
> pdftex testeol-cr
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=pdftex)
 restricted \write18 enabled.
entering extended mode
(./testeol-cr.tex [1{/usr/local/texlive/2017/texmf-var/fonts/map/pdftex/updmap/
pdftex.map}] )</usr/local/texlive/2017/texmf-dist/fonts/type1/public/amsfonts/c
m/cmr10.pfb>
Output written on testeol-cr.pdf (1 page, 10842 bytes).
Transcript written on testeol-cr.log.
> pdftex testeol-crlf
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=pdftex)
 restricted \write18 enabled.
entering extended mode
(./testeol-crlf.tex [1{/usr/local/texlive/2017/texmf-var/fonts/map/pdftex/updma
p/pdftex.map}] )</usr/local/texlive/2017/texmf-dist/fonts/type1/public/amsfonts
/cm/cmr10.pfb>
Output written on testeol-crlf.pdf (1 page, 10842 bytes).
Transcript written on testeol-crlf.log.

This should make clear that the TeX engines are set up to look for the record terminator, to “guess” it and to act consequently. Indeed, the three PDF files only differ for time stamps.

As far as the translation into ASCII, TeX uses two internal arrays called xord and xchr (see tex.web) for more information. Essentially, xord defines the translation from input to ASCII for internal use, whereas xchr defines the translation from the internal ASCII to whatever one wants (but it is initialized as the identity map).

The xord array can be changed at runtime by using TCX files (see section 4.4.2 in texdoc web2c for more information). EBCDIC (or other encoding) based implementation need to use the change file for defining the correct xord array.

Some TeX extensions, notably MLTeX and encTeX, allow for changing the xchr array in various ways (see texdoc web2c for MLTeX and texdoc enctex for encTeX).