The TeX engine processes the input file on a line-by-line basis (The TEXbook, p. 46). The first line will be read from the file, undergo some pre-processing (see below), and then the engine will start processing it. When the line has been processed, the next line will be read from the file, etc.
The procedure that reads the next line from the file is effectively separate and independent from the TeX engine; in particular, it is "oblivious" to the concept of category codes, and has effectively no access to the catcode table. Therefore, as far as this procedure is concerned, a line of input is defined the way the operating system defines it. For instance, on a Windows system the end of a line is indicated with a CR-LF combination, whereas on a Linux system it is indicated with LF. The end-of-line marker, if any, is not included in the list of bytes returned by the reading procedure (TeX by Topic, p. 29).
After the line is read in, and before the engine begins to process it, the line undergoes some pre-processing:
- (The TEXbook, p. 43) The characters are converted from whatever encoding they were represented in inside the input file to the ASCII encoding. For instance, if the file was encoded with EBCDIC, then a byte representing the number 129 (which is EBCDIC's code-point for lowercase a), will be replaced by a byte representing the number 97 (which is ASCII's code-point for lowercase a).
- (The TEXbook, p. 46) Any trailing spaces (ASCII 32) are dropped.
- (The TEXbook, p. 46) A Carriage Return (ASCII 13) is appended at the end of the line.
The line is now ready to be processed by the TeX engine, and is handed over to the tokenizer.
As the description above shows, the TeX engine depends on the following parameters:
- The OS's end-of-line marker
- The input file's encoding
and cannot even begin to process the input file without knowing their values.
Are these parameters customizable by the user, or are they hard-coded in the source code? If it is the former, how can they be customized for, say, pdftex?
If it is the latter, in what sense are they hard-coded:
- In the sense that every distribution is built from a slightly different source code customized to a certain end-of-line marker/file encoding configuration?
- In the sense that the source code tests for these values during run-time?
- In the sense that the source code uses some portable API for reading lines and characters?
- Knuth, Donald Ervin. The TEXbook: Addison-Wesley, 1991.
- Eijkhout, Victor. TeX by Topic: Addison-Wesley, 1992.