4

Why does LaTeX (TeX Live 2020, Debian) tell me that SomeLines.txt has a 6th line which contains the token \par?

I know with \endlinechar being 13 and the catcode of char 13 (carriage return) being 5 you get \par for an empty line because a carriage-return character is appended to the empty line and that carriage-return character is processed while the carriage-return character is of category 5(end of line) and TeX's reading-apparatus is in state N.

So I suppose TeX's routine for reading a line of input "thinks" there is an empty line/an empty record after Line 5 and appends the endline-character, whose category is 5(end of line). When processing that character, TeX's reading-apparatus is in state N leading to the appended endline-character yielding the control-word-token \par.

If so:

  • Why does TeX think that there is an empty line/an empty record after Line 5?
  • Do the routines for reading a line of input of all TeX distributions think so?
\begin{filecontents*}{SomeLines.txt}
Line1
Line2
Line3
Line4
Line5
\end{filecontents*}

\newread\SomeLinesRead
\newcount\SomeLinesCount
\def\CountLinesLoop{%
  \ifeof\SomeLinesRead
  \else
     \immediate\read\SomeLinesRead to \Thisline
     \global\advance\SomeLinesCount by 1 %
     \message{^^JLine \number\SomeLinesCount="\Thisline"}%
  \expandafter\CountLinesLoop\fi
}
\immediate\openin\SomeLinesRead SomeLines.txt
\SomeLinesCount=0
\begingroup%
% Changes of the value of \endlinechar are reflected in the line "Line6=...".
% If \endlinechar denotes the number of the code-point of a
% character of category 5(end of line), the state of the reading-
% apparatus is reflected, too, which apparently is N when the
% character appended to line/record 6 is processed.
%\endlinechar=`\A  % -> Line 6="A" 
%\endlinechar=`\B  % -> Line 6="B" 
\CountLinesLoop%
\endgroup%
\immediate\closein\SomeLinesRead
\message{^^JTotal amount of lines: \number\SomeLinesCount}%
\stop

Console output:

This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode
(./test.tex
LaTeX2e <2020-10-01> patch level 4
L3 programming layer <2021-02-18>

LaTeX Warning: Writing file `./SomeLines.txt'.



Line 1="Line1 " 
Line 2="Line2 " 
Line 3="Line3 " 
Line 4="Line4 " 
Line 5="Line5 " 
Line 6="\par " 
Total amount of lines: 6 )
No pages of output.
Transcript written on test.log.

I assume:

The condition for terminating the loop is not "if the last record-terminator of the file is reached".

The condition for terminating the loop is \ifeof, i.e., "if the end of the file is reached".

There are five lines/records (none of them containing unmatched characters of category 1(begin group) so that in this scenario a single \read-operation processes a single line/record) and thus five "record-terminators" and one "end of file" belonging to SomeLines.txt, which makes a total of six reading-iterations until the end of the file is reached. In the 6th iteration the "line/record" is considered empty because there is nothing between the last "record-terminator" and the "end of file".
TeX's handling of empty lines and the coming into being of the token \par is described above.

Is this interpretation correct?


Addendum in November 3, 2021:

In his answer David Carlisle pointed out:

tex.web 9507 says

@ An empty line is appended at the end of a |read_file|. @^empty
line at end of file@> 

Seems this means an empty line/record is appended.
Seems this is done on purpose.

Thus subsequent questions are:

Why? What purpose?

Did I overlook the documentation of this "feature" in the TeXbook?

Is this one of the many features for which you need to study tex.web to become aware?


Addendum in January 20, 2022:

The TeXbook says in Chapter 20: Definitions (also called Macros):

To get input from an open file, you say
\read⟨number⟩to⟨control sequence⟩
and the control sequence is defined to be a parameterless macro whose replacement text is the contents of the next line read from the designated file. This line is converted to a token list, using the procedure of Chapter 8, based on the current category codes.
Additional lines are read, if necessary, until an equal number of left and right braces has been found. An empty line is implicitly appended to the end of a file that is being \read.

5
  • 3
    tex is really record based and doesn't assume the input has eol characters at all, the last line whether terminated by eol or eof in the file is normalised to the same tex input Nov 2, 2021 at 20:43
  • 1
    if you set \endlinechar to -1 the final "fake line" is shown as empty, but I'm not sure if that answers your question Nov 2, 2021 at 20:59
  • @DavidCarlisle I edited my question and called the thing "record terminator". The question could be rephrased: Why dies TeX assume the presence of an empty record after the record "Line5"? Nov 2, 2021 at 21:01
  • @DavidCarlisle I know. I kept \endlinechar's value 13 because appending \endlinechar is another action that takes place on record-basis and the resulting \par shows that there must have been an empty record where the endline-character was appended to. (Otherwise the reading-apparatus would not have been in state N leading to the endline-character yielding \par.) If TeX did not assume an empty record, it would not have appended the endline-character... Nov 2, 2021 at 21:06
  • 3
    yes as I show below, tex just explicitly (and somewhat arbitrarily) acts as if there is a blank line at the end, Probably some pre-historic normalisation of interesting file systems of the day Nov 2, 2021 at 21:09

1 Answer 1

5

When reading files tex acts essentially the same way whether or not there is an end of line character at the end of the file. The final line is taken as a line and \endlinechar is added to the end of each line if this is set to a legal character value.

tex.web 9507 says

@ An empty line is appended at the end of a |read_file|.
@^empty line at end of file@>

so if \endlinechar is the normal 13 and character 13 has catcode 5 then as usual that ends up being reported as \par. If you set \endlinechar=-1 then the final line before the \ifeof is true will be reported as empty.

1
  • Many thanks for pointing to "@ An empty line is appended at the end of a |read_file|." Seems this means an empty record is appended. So this is done on purpose. Now the question is: Why? What purpose? :-) | Now I noticed: You answered this in your other comment. Thank you. Nov 2, 2021 at 21:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.