The effect of triggering the re-writing/creating anew of a text file which currently is processed via `\input` puzzles me?

Question

I tried to create a .tex-file fileb.tex which can be \input by filea.tex and which - when being input - stops being input and triggers rewriting itself.

In my first attempts I erroneously omitted \endinput.

Hereby I stumbled over some behavior which I do not expect for the case of omitting \endinput:

This is filea.tex:

\newwrite\filebwrite
%
% Create the initial fileb.tex:
\immediate\openout\filebwrite fileb.tex %
\immediate\write\filebwrite{This is fileb.}%
\immediate\write\filebwrite{{\string\tt\string\string\string\macro} was not used when writing it.}%
\immediate\write\filebwrite{\string\recreatefileb}%
\immediate\write\filebwrite{Something in fileb.tex that you should not see in the pdf.}%
\immediate\closeout\filebwrite
%
%
\def\recreatefileb{%
  \immediate\openout\filebwrite fileb.tex %
  \immediate\write\filebwrite{This is the re-written fileb.}%
  \immediate\write\filebwrite{At the time of writing it {\string\tt\string\string\string\macro} expanded to: \macro}%
  \immediate\write\filebwrite{\string\recreatefileb}%
  \immediate\write\filebwrite{Something in fileb.tex that you should not see in the pdf.}%
  \immediate\closeout\filebwrite
}%
%
fileb.tex is now processed.

\def\macro{Rewrite 1}%
\input fileb.tex % Here you should have: ... was not used when writing it.

\def\macro{Rewrite 2}%
\input fileb.tex % Here you should have: ... expanded to: Rewrite 1

\def\macro{Rewrite 3}%
\input fileb.tex % Here you should have: ... expanded to: Rewrite 2
%
% When looking at fileb.tex after compiling filea.tex, you should see:
%
%  This is the re-written fileb.
%  At the time of writing it {\tt\string\macro} expanded to: Rewrite 3
%  \recreatefileb
%  Something in fileb.tex that you should not see in the pdf.
%
\bye

When I compile filea.tex, I get the following output:

Also fileb.tex is created and re-created during compilation of filea.tex. fileb.tex exists after compiling filea.tex and looks as expected by me:

This is the re-written fileb.
At the time of writing it {\tt\string\macro} expanded to: Rewrite 3
\recreatefileb
Something in fileb.tex that you should not see in the pdf.

I don't get any error-messages at all.

filea.log looks like this:

This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2019/dev/Debian) (preloaded format=pdftex 2020.2.13)  9 AUG 2020 11:50
entering extended mode
 \write18 enabled.
 %&-line parsing enabled.
**filea.tex
(./filea.tex
\filebwrite=\write0
\openout0 = `fileb.tex'.

 (./fileb.tex
\openout0 = `fileb.tex'.

) (./fileb.tex
\openout0 = `fileb.tex'.

) (./fileb.tex
\openout0 = `fileb.tex'.

) [1{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map}] )</usr/share/texlive/t
exmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb></usr/share/texlive/texmf-di
st/fonts/type1/public/amsfonts/cm/cmtt10.pfb>
Output written on filea.pdf (1 page, 26042 bytes).
PDF statistics:
 16 PDF objects out of 1000 (max. 8388607)
 10 compressed objects within 1 object stream
 0 named destinations out of 1000 (max. 500000)
 1 words of extra memory for PDF output out of 10000 (max. 10000000)

The terminal shows this:

$ pdftex -shell-escape filea.tex
This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2019/dev/Debian) (preloaded format=pdftex)
 \write18 enabled.
entering extended mode
(./filea.tex (./fileb.tex) (./fileb.tex) (./fileb.tex) [1{/var/lib/texmf/fonts/
map/pdftex/updmap/pdftex.map}] )</usr/share/texlive/texmf-dist/fonts/type1/publ
ic/amsfonts/cm/cmr10.pfb></usr/share/texlive/texmf-dist/fonts/type1/public/amsf
onts/cm/cmtt10.pfb>
Output written on filea.pdf (1 page, 26042 bytes).
Transcript written on filea.log.

I am confused and I have the following questions:

1. Why is there the phrase at you should not see in the pdf. (marked in red color in the picture above) in the .pdf-output-file?
   You are often told that TeX's \input does process files line by line. (TeX's "eyes" look at a line of input and pass the characters of that line to the "mouth" for tokenization...)
   But it seems some back part of the line of input where that phrase occurs is processed (twice).
   It is surprising to me that it looks like something goes through the "mouth" of TeX which is not the characters of a complete \input-line but which is only the characters of some back part of a line of input.

2. At the time of re-writing fileb.tex, no \endinput occured, thus fileb.tex is still open and used for reading. Why does this not yield an error-message? Why does it seem to be possible to re-write a file while it is used as the input-file?

3. Why are there so many phrases Something in fileb.tex that you should not see in the pdf. in the .pdf-file at all? In fileb.tex this phrase always occurs on a line on its own. And I assumed that line is not processed yet/is not in TeX's mouth yet when \recreatefileb triggers creating fileb.tex anew.

---

By the way: The following seems to do what I have in mind:

filea.tex:

\newwrite\filebwrite
%
% Create the initial fileb.tex:
\immediate\openout\filebwrite fileb.tex %
\immediate\write\filebwrite{This is fileb.}%
\immediate\write\filebwrite{{\string\tt\string\string\string\macro} was not used when writing it.}%
\immediate\write\filebwrite{\string\expandafter\string\endinput}%
\immediate\write\filebwrite{\string\recreatefileb}%
\immediate\write\filebwrite{Something in fileb.tex that you should not see in the pdf.}%
\immediate\closeout\filebwrite
%
%
\def\recreatefileb{%
  \immediate\openout\filebwrite fileb.tex %
  \immediate\write\filebwrite{This is the re-written fileb.}%
  \immediate\write\filebwrite{At the time of writing it {\string\tt\string\string\string\macro} expanded to: \macro}%
  \immediate\write\filebwrite{\string\expandafter\string\endinput}%
  \immediate\write\filebwrite{\string\recreatefileb}%
  \immediate\write\filebwrite{Something in fileb.tex that you should not see in the pdf.}%
  \immediate\closeout\filebwrite
}%
%
fileb.tex is now processed.

\def\macro{Rewrite 1}%
\input fileb.tex % Here you should have: ... was not used when writing it.

\def\macro{Rewrite 2}%
\input fileb.tex % Here you should have: ... expanded to: Rewrite 1

\def\macro{Rewrite 3}%
\input fileb.tex % Here you should have: ... expanded to: Rewrite 2
%
% When looking at fileb.tex after compiling filea.tex, you should see:
%
%  This is the re-written fileb.
%  At the time of writing it {\tt\string\macro} expanded to: Rewrite 3
%  \expandafter\endinput
%  \recreatefileb
%  Something in fileb.tex that you should not see in the pdf.
%
\bye

The output as expected:

But even here I am still a little confused at the moment:

The definition of \endinput from page 214 of The TeXbook is:

\endinput. The expansion is null. The next time TeX gets to the end of an \input line, it will stop reading from the file containing that line.

In fileb.tex (beneath other things) you have two input-lines:

\expandafter\endinput
\recreatefileb

Which of these lines is considered the line where "the next time after processing \endinput TeX gets to the end of an \input line"?

I assume:

Due to \expandafter the token \recreatefileb (and thus also the end of the line which contains the \endinput) is processed before processing \endinput.

So the next time after processing \endinput TeX gets to the end of an \input-line after having processed \recreatefileb, when reaching the end of the line which contains the string \recreatefileb. Thus the subsequent line Something in fileb.tex that you should not see in the pdf. is not processed. (It would also not be processed if \recreateb was just a no-op.)

Are my assumptions correct?

If so, the following question arises:

At the time when \recreatefileb triggers re-writing fileb.tex, \endinput is not processed yet. Thus fileb.tex is still to be considered the \input-file to read from. Why is it possible to re-write a file although it is considered the \input-file to read from?

If not so, the following question arises:

Where am I wrong?

Answer 1

I will assume a roughly POSIX like operating system (aka. Windows is weird and might behave differently).

Let's start with your question 2: There are often good usecases for reading files while someone else is writing it, for example this allows live monitoring of log files (any efficient implementation of tail -f will do this), primitive inter process communication and similar. On the other side, there are no good reasons why it should fail. If an application can't deal with it, it can always use some locking systems.

For the other points, you have to know that "reading a line of input from file" is an abstraction implemented by the standard library on top of system calls which have no notion of a "line". A file on that level is just a bunch of bytes.

So the request to "Read one line of input" is implemented by keeping an internal cache of the next 500-ish bytes of the files, which i always scanned for the next end of a line. If it is found, the line is returned, otherwise the next bytes are read from the file.

So when the file is recreated, the last line of the original file is often still in the cache and returned without looking at the actual file at all. Only when the end of the file is reached, the library asks the operating system if there is some additional content. Remember: At this level, there are no lines, only bytes. So if the files used to be 420 bytes long, the library asks: "Can you read some bytes starting at position 421?" If the file has been changed, this offset might now be in the middle of some line. Therefore the system successfully returns the rest of the file, and the library returns this as the next line (after all, it had read a newline character at offset 420, so 421 "obviously" starts a new line).

The whole issue shows that these systems can deal with changing files, but they assume that you only append additional content, don't change existing stuff.

By the way, this is specific to changing the existing file. A common method e.g. used by text editors to change a file is to delete the old file and create a new file with the old name (it's a bit more complicated, but it's the general idea). In this case, the file you still read would still be the deleted old file and therefore you would not pick up random stuff from the new file. (You would more reliably be able to read the remaining parts of the old file though)