How TeX reads what you type

Question

These questions are connected with this answer: Carriage-return and line-feed in web2cTeX

Suppose TeX is reading an input file and is at the beginning of new line, in state N. New end-of-line character must be converted to \par token. But this happens after the line has been read from the file (i.e, not in the "eyes" of TeX), right? I mean, this substitution cannot happen in readln procedure, can it? If not, in what part of TeX does this substitution take place?

One more question: if end-of-line character is appended to each line buffer, does this mean that end-of-line character is before \par?

I explain myself:

Suppose TeX has just read a line from input file. Now there is end-of-line character at the end of the line buffer. Now TeX reads an empty line and converts it to \par. So TeX now has end-of-line character, and \par token in its stomach, right? I mean, is end-of-line character, followed by \par, removed, or it just stays there and does not influence the output in any way?

Answer 1

The system dependent "end-of-record" is determined as explained in Carriage-return and line-feed in web2cTeX (at least in Web2C implementations).

Now the rule explained on page 46 of the TeXbook applies:

TeX deletes any ⟨space⟩ characters (number 32) that occur at the right end of an input line. Then it inserts a ⟨return⟩ character (number 13) at the right end of the line, except that it places nothing additional at the end of a line that you inserted with ‘I’ during error recovery. Note that ⟨return⟩ is considered to be an actual character that is part of the line; you can obtain special effects by changing its catcode.

This should be supplemented by the fact that Web2C implementations also strip off ⟨tab⟩ characters (number 9) and the paragraph on page 48 of the TeXbook:

The special character inserted at the end of each line needn’t be ⟨return⟩; TeX actually inserts the current value of an integer parameter called \endlinechar, which normally equals 13 but it can be changed like any other parameter. If the value of \endlinechar is negative or greater than 255, no character is appended, and the effect is as if every line ends with % (i.e., with a comment character).

^{On LuaTeX and XeLaTeX, \endlinechar is allowed to be up to "10FFFF, that is, 1114111.}

The “system dependent end-of-record” is no more relevant when the substitution above has been performed. Note that a change to \endlinechar will not be reflected on the current line, because lines are read (not tokenized) one by one as soon as need occurs. For instance, this code will result in three P's after the period, but no P before ‘x’.

This text will not have a P \endlinechar=`P before
x.

\endlinechar=13

\bye

Note also that only a category code character found in state N will trigger insertion of \par.