# How is the TeX engine informed of the OS's end-of-line marker and file encoding?

The TeX engine processes the input file on a line-by-line basis (The TEXbook, p. 46). The first line will be read from the file, undergo some pre-processing (see below), and then the engine will start processing it. When the line has been processed, the next line will be read from the file, etc.

The procedure that reads the next line from the file is effectively separate and independent from the TeX engine; in particular, it is "oblivious" to the concept of category codes, and has effectively no access to the catcode table. Therefore, as far as this procedure is concerned, a line of input is defined the way the operating system defines it. For instance, on a Windows system the end of a line is indicated with a CR-LF combination, whereas on a Linux system it is indicated with LF. The end-of-line marker, if any, is not included in the list of bytes returned by the reading procedure (TeX by Topic, p. 29).

After the line is read in, and before the engine begins to process it, the line undergoes some pre-processing:

1. (The TEXbook, p. 43) The characters are converted from whatever encoding they were represented in inside the input file to the ASCII encoding. For instance, if the file was encoded with EBCDIC, then a byte representing the number 129 (which is EBCDIC's code-point for lowercase a), will be replaced by a byte representing the number 97 (which is ASCII's code-point for lowercase a).
2. (The TEXbook, p. 46) Any trailing spaces (ASCII 32) are dropped.
3. (The TEXbook, p. 46) A Carriage Return (ASCII 13) is appended at the end of the line.

The line is now ready to be processed by the TeX engine, and is handed over to the tokenizer.

As the description above shows, the TeX engine depends on the following parameters:

• The OS's end-of-line marker
• The input file's encoding

and cannot even begin to process the input file without knowing their values.

1. Are these parameters customizable by the user, or are they hard-coded in the source code? If it is the former, how can they be customized for, say, pdftex?

2. If it is the latter, in what sense are they hard-coded:

1. In the sense that every distribution is built from a slightly different source code customized to a certain end-of-line marker/file encoding configuration?
2. In the sense that the source code tests for these values during run-time?
3. In the sense that the source code uses some portable API for reading lines and characters?

Cited references

• Knuth, Donald Ervin. The TEXbook: Addison-Wesley, 1991.
• Eijkhout, Victor. TeX by Topic: Addison-Wesley, 1992.
• This is done in the change file for the specific OS (at compile time). With TeX Live, based on Web2C, the end-of-record marker is set on a file specific way, so essentially any of the conventions regarding the end-of-record marker can be used, be it `<LF>`, `<CR>` or `<CR><LF>`. – egreg Aug 28 '17 at 8:58
• Web2C based engines (such as TeX Live or MiKTeX) can also use `tcx` files for interacting with the xord and xchr internal arrays, see section 4.4.2 in `texdoc web2c` for more information. – egreg Aug 28 '17 at 9:03
• For historical completeness, some file systems (for example fixed length record files on IBM's mainframe operating systems OS/360 and OS/370 had no end-of-line markers embedded anywhere in the data. That's why TeX needed to "drop trailing spaces, and add a CR at the end of each line" - to convert the file contents to match the way a human would read a printed copy or an on-screen display of the file. There were other file formats on OS/360 that did have a variety of different types of "end of record marker" so it made sense to convert all input into a standard form as early as possible. – alephzero Aug 28 '17 at 10:41

The record terminator is OS dependent, but Web2C based TeX distributions (TeX Live and MiKTeX) can decide it on the fly. I created three identical files containing

``````abc
def
\bye
``````

but with different line terminators; here are the hex dumps:

``````> hexdump testeol-lf.tex
0000000 61 62 63 0a 64 65 66 0a 5c 62 79 65 0a
000000d
> hexdump testeol-cr.tex
0000000 61 62 63 0d 64 65 66 0d 5c 62 79 65 0d
000000d
> hexdump testeol-crlf.tex
0000000 61 62 63 0d 0a 64 65 66 0d 0a 5c 62 79 65 0d 0a
0000010
``````

If I compile them, I get

``````> pdftex testeol-lf
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=pdftex)
restricted \write18 enabled.
entering extended mode
(./testeol-lf.tex [1{/usr/local/texlive/2017/texmf-var/fonts/map/pdftex/updmap/
pdftex.map}] )</usr/local/texlive/2017/texmf-dist/fonts/type1/public/amsfonts/c
m/cmr10.pfb>
Output written on testeol-lf.pdf (1 page, 10842 bytes).
Transcript written on testeol-lf.log.
> pdftex testeol-cr
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=pdftex)
restricted \write18 enabled.
entering extended mode
(./testeol-cr.tex [1{/usr/local/texlive/2017/texmf-var/fonts/map/pdftex/updmap/
pdftex.map}] )</usr/local/texlive/2017/texmf-dist/fonts/type1/public/amsfonts/c
m/cmr10.pfb>
Output written on testeol-cr.pdf (1 page, 10842 bytes).
Transcript written on testeol-cr.log.
> pdftex testeol-crlf
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=pdftex)
restricted \write18 enabled.
entering extended mode
(./testeol-crlf.tex [1{/usr/local/texlive/2017/texmf-var/fonts/map/pdftex/updma
p/pdftex.map}] )</usr/local/texlive/2017/texmf-dist/fonts/type1/public/amsfonts
/cm/cmr10.pfb>
Output written on testeol-crlf.pdf (1 page, 10842 bytes).
Transcript written on testeol-crlf.log.
``````

This should make clear that the TeX engines are set up to look for the record terminator, to “guess” it and to act consequently. Indeed, the three PDF files only differ for time stamps.

As far as the translation into ASCII, TeX uses two internal arrays called xord and xchr (see `tex.web`) for more information. Essentially, xord defines the translation from input to ASCII for internal use, whereas xchr defines the translation from the internal ASCII to whatever one wants (but it is initialized as the identity map).

The xord array can be changed at runtime by using TCX files (see section 4.4.2 in `texdoc web2c` for more information). EBCDIC (or other encoding) based implementation need to use the change file for defining the correct xord array.

Some TeX extensions, notably MLTeX and encTeX, allow for changing the xchr array in various ways (see `texdoc web2c` for MLTeX and `texdoc enctex` for encTeX).

• Does the `inputenc` package work by changing xord during runtime? – Evan Aad Aug 30 '17 at 9:53
• @EvanAad No. It works with active characters. – egreg Aug 30 '17 at 9:54
• @EvanAad It would be impossible to load a TCX file after the engine (pdftex or whatever) has started: the TCX file can only be loaded with a command line option. – egreg Aug 30 '17 at 10:07