# The effect of triggering the re-writing/creating anew of a text file which currently is processed via \input puzzles me?

I tried to create a .tex-file fileb.tex which can be \input by filea.tex and which - when being input - stops being input and triggers rewriting itself.

In my first attempts I erroneously omitted \endinput.

Hereby I stumbled over some behavior which I do not expect for the case of omitting \endinput:

This is filea.tex:

\newwrite\filebwrite
%
% Create the initial fileb.tex:
\immediate\openout\filebwrite fileb.tex %
\immediate\write\filebwrite{This is fileb.}%
\immediate\write\filebwrite{{\string\tt\string\string\string\macro} was not used when writing it.}%
\immediate\write\filebwrite{\string\recreatefileb}%
\immediate\write\filebwrite{Something in fileb.tex that you should not see in the pdf.}%
\immediate\closeout\filebwrite
%
%
\def\recreatefileb{%
\immediate\openout\filebwrite fileb.tex %
\immediate\write\filebwrite{This is the re-written fileb.}%
\immediate\write\filebwrite{At the time of writing it {\string\tt\string\string\string\macro} expanded to: \macro}%
\immediate\write\filebwrite{\string\recreatefileb}%
\immediate\write\filebwrite{Something in fileb.tex that you should not see in the pdf.}%
\immediate\closeout\filebwrite
}%
%
fileb.tex is now processed.

\def\macro{Rewrite 1}%
\input fileb.tex % Here you should have: ... was not used when writing it.

\def\macro{Rewrite 2}%
\input fileb.tex % Here you should have: ... expanded to: Rewrite 1

\def\macro{Rewrite 3}%
\input fileb.tex % Here you should have: ... expanded to: Rewrite 2
%
% When looking at fileb.tex after compiling filea.tex, you should see:
%
%  This is the re-written fileb.
%  At the time of writing it {\tt\string\macro} expanded to: Rewrite 3
%  \recreatefileb
%  Something in fileb.tex that you should not see in the pdf.
%
\bye

When I compile filea.tex, I get the following output:

Also fileb.tex is created and re-created during compilation of filea.tex. fileb.tex exists after compiling filea.tex and looks as expected by me:

This is the re-written fileb.
At the time of writing it {\tt\string\macro} expanded to: Rewrite 3
\recreatefileb
Something in fileb.tex that you should not see in the pdf.

I don't get any error-messages at all.

filea.log looks like this:

This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2019/dev/Debian) (preloaded format=pdftex 2020.2.13)  9 AUG 2020 11:50
entering extended mode
\write18 enabled.
%&-line parsing enabled.
**filea.tex
(./filea.tex
\filebwrite=\write0
\openout0 = fileb.tex'.

(./fileb.tex
\openout0 = fileb.tex'.

) (./fileb.tex
\openout0 = fileb.tex'.

) (./fileb.tex
\openout0 = fileb.tex'.

) [1{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map}] )</usr/share/texlive/t
exmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb></usr/share/texlive/texmf-di
st/fonts/type1/public/amsfonts/cm/cmtt10.pfb>
Output written on filea.pdf (1 page, 26042 bytes).
PDF statistics:
16 PDF objects out of 1000 (max. 8388607)
10 compressed objects within 1 object stream
0 named destinations out of 1000 (max. 500000)
1 words of extra memory for PDF output out of 10000 (max. 10000000)

The terminal shows this:

\$ pdftex -shell-escape filea.tex
This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2019/dev/Debian) (preloaded format=pdftex)
\write18 enabled.
entering extended mode
(./filea.tex (./fileb.tex) (./fileb.tex) (./fileb.tex) [1{/var/lib/texmf/fonts/
map/pdftex/updmap/pdftex.map}] )</usr/share/texlive/texmf-dist/fonts/type1/publ
ic/amsfonts/cm/cmr10.pfb></usr/share/texlive/texmf-dist/fonts/type1/public/amsf
onts/cm/cmtt10.pfb>
Output written on filea.pdf (1 page, 26042 bytes).
Transcript written on filea.log.

I am confused and I have the following questions:

1. Why is there the phrase at you should not see in the pdf. (marked in red color in the picture above) in the .pdf-output-file?
You are often told that TeX's \input does process files line by line. (TeX's "eyes" look at a line of input and pass the characters of that line to the "mouth" for tokenization...)
But it seems some back part of the line of input where that phrase occurs is processed (twice).
It is surprising to me that it looks like something goes through the "mouth" of TeX which is not the characters of a complete \input-line but which is only the characters of some back part of a line of input.

2. At the time of re-writing fileb.tex, no \endinput occured, thus fileb.tex is still open and used for reading. Why does this not yield an error-message? Why does it seem to be possible to re-write a file while it is used as the input-file?

3. Why are there so many phrases Something in fileb.tex that you should not see in the pdf. in the .pdf-file at all? In fileb.tex this phrase always occurs on a line on its own. And I assumed that line is not processed yet/is not in TeX's mouth yet when \recreatefileb triggers creating fileb.tex anew.

By the way: The following seems to do what I have in mind:

filea.tex:

\newwrite\filebwrite
%
% Create the initial fileb.tex:
\immediate\openout\filebwrite fileb.tex %
\immediate\write\filebwrite{This is fileb.}%
\immediate\write\filebwrite{{\string\tt\string\string\string\macro} was not used when writing it.}%
\immediate\write\filebwrite{\string\expandafter\string\endinput}%
\immediate\write\filebwrite{\string\recreatefileb}%
\immediate\write\filebwrite{Something in fileb.tex that you should not see in the pdf.}%
\immediate\closeout\filebwrite
%
%
\def\recreatefileb{%
\immediate\openout\filebwrite fileb.tex %
\immediate\write\filebwrite{This is the re-written fileb.}%
\immediate\write\filebwrite{At the time of writing it {\string\tt\string\string\string\macro} expanded to: \macro}%
\immediate\write\filebwrite{\string\expandafter\string\endinput}%
\immediate\write\filebwrite{\string\recreatefileb}%
\immediate\write\filebwrite{Something in fileb.tex that you should not see in the pdf.}%
\immediate\closeout\filebwrite
}%
%
fileb.tex is now processed.

\def\macro{Rewrite 1}%
\input fileb.tex % Here you should have: ... was not used when writing it.

\def\macro{Rewrite 2}%
\input fileb.tex % Here you should have: ... expanded to: Rewrite 1

\def\macro{Rewrite 3}%
\input fileb.tex % Here you should have: ... expanded to: Rewrite 2
%
% When looking at fileb.tex after compiling filea.tex, you should see:
%
%  This is the re-written fileb.
%  At the time of writing it {\tt\string\macro} expanded to: Rewrite 3
%  \expandafter\endinput
%  \recreatefileb
%  Something in fileb.tex that you should not see in the pdf.
%
\bye

The output as expected:

But even here I am still a little confused at the moment:

The definition of \endinput from page 214 of The TeXbook is:

\endinput. The expansion is null. The next time TeX gets to the end of an \input line, it will stop reading from the file containing that line.

In fileb.tex (beneath other things) you have two input-lines:

\expandafter\endinput
\recreatefileb

Which of these lines is considered the line where "the next time after processing \endinput TeX gets to the end of an \input line"?

I assume:

Due to \expandafter the token \recreatefileb (and thus also the end of the line which contains the \endinput) is processed before processing \endinput.

So the next time after processing \endinput TeX gets to the end of an \input-line after having processed \recreatefileb, when reaching the end of the line which contains the string \recreatefileb. Thus the subsequent line Something in fileb.tex that you should not see in the pdf. is not processed. (It would also not be processed if \recreateb was just a no-op.)

Are my assumptions correct?

If so, the following question arises:

At the time when \recreatefileb triggers re-writing fileb.tex, \endinput is not processed yet. Thus fileb.tex is still to be considered the \input-file to read from. Why is it possible to re-write a file although it is considered the \input-file to read from?

If not so, the following question arises:

Where am I wrong?

• I'm not sure that this is tex-defined behaviour, if you write to a file that is open for reading are you not at the mercy of operating system level file buffering? Aug 9, 2020 at 10:23
• A complication here is that it's not entirely obvious that it is the same file. the \openin will find any file of that name in the input path, and the \openout will write to --output-directory if that is specified. So it may be that neither is referencing a file of that name in the current directory. Aug 9, 2020 at 11:37
• I don't suppose a response of --output-directory is always a request for self-inflicted pain, so breakage there is expected, is the response that you want? I suppose you'd need to look in texk/web2c/lib/openclose.c (which already has enough comments about complications caused bu output-directory:-) Aug 9, 2020 at 11:47
• @DavidCarlisle Maybe. But BASIC interpreters known to me in the 90s of the last century raised error-messages with attempts to (re-)write (to) a file that was open for reading. Same with programs compiled via Borland's Turbo Pascal. Maybe underlying routines on os-level were used for checking whether the file was open for reading. The error-messages came from the compiler/from routines inserted into the exexutable by the pascal-compiler. With TeX you can probably use \ifeof for implementing a checking-routine yourself. But the behaviour if you don't is something I don't like. ;-) Aug 9, 2020 at 11:48
• I really don't want to trace the file open code here (You could ask Karl on texlive list) but I assume that the file pointer is essentially corrupted so you are just seeing broken output. Aug 9, 2020 at 12:01

I will assume a roughly POSIX like operating system (aka. Windows is weird and might behave differently).

Let's start with your question 2: There are often good usecases for reading files while someone else is writing it, for example this allows live monitoring of log files (any efficient implementation of tail -f will do this), primitive inter process communication and similar. On the other side, there are no good reasons why it should fail. If an application can't deal with it, it can always use some locking systems.

For the other points, you have to know that "reading a line of input from file" is an abstraction implemented by the standard library on top of system calls which have no notion of a "line". A file on that level is just a bunch of bytes.

So the request to "Read one line of input" is implemented by keeping an internal cache of the next 500-ish bytes of the files, which i always scanned for the next end of a line. If it is found, the line is returned, otherwise the next bytes are read from the file.

So when the file is recreated, the last line of the original file is often still in the cache and returned without looking at the actual file at all. Only when the end of the file is reached, the library asks the operating system if there is some additional content. Remember: At this level, there are no lines, only bytes. So if the files used to be 420 bytes long, the library asks: "Can you read some bytes starting at position 421?" If the file has been changed, this offset might now be in the middle of some line. Therefore the system successfully returns the rest of the file, and the library returns this as the next line (after all, it had read a newline character at offset 420, so 421 "obviously" starts a new line).

The whole issue shows that these systems can deal with changing files, but they assume that you only append additional content, don't change existing stuff.

By the way, this is specific to changing the existing file. A common method e.g. used by text editors to change a file is to delete the old file and create a new file with the old name (it's a bit more complicated, but it's the general idea). In this case, the file you still read would still be the deleted old file and therefore you would not pick up random stuff from the new file. (You would more reliably be able to read the remaining parts of the old file though)

• Ad There are often good usecases for reading files while someone else is writing it, for example this allows live monitoring of log files: Yes, monitoring .log-files is nice. Usually a .log-file is not processed by a compiler in the same way in which a .tex-\input-file is.I think that it should not be possible to (re)write (to) a file while that file is compiled. And "compiling" is the situation when processing via \input. Thus the file currently compiled/currently processed by TeX should be locked for writing. Attempts at writing to it during compilation should yield error-messages. Aug 10, 2020 at 9:28
• @UlrichDiez Well, in most cases, it simply doesn't matter because editors use the delete and recreate technique anyway. If the file would be locked for writing, thi might no longer workwhich would led to major issues. (You would not be able to change anything in your file until your compiler finished.) Also even for TeX sources it is not clear that this is always the right behaviour. It is probably uncommon to want to write to normal files, but e.g. if a named pipe is used as input, writing into it while TeX is reading it makes a lot of sense for typesetting streamed data. Aug 10, 2020 at 9:34
• For a good example how such locking leads to issues, see Adobe Reader on Windows: While a PDF file is opened in Adobe Reader is is locked, therefore you can't compile until you close the viewer... Aug 10, 2020 at 9:35
• Why should one change a .tex-source while the compiler is compiling it? That does not make sense to me. Do you change the C-source-code of a program while the C-compiler is creating the binary? When programming in TeX I usually have the file open in the editor. When I click on the editor's "save"-button, the modified file is stored. Doing this should not be possible while the compiler processes that file. Saving the modified file while the compiler is not running/processing it, is okay. But: Modifying the source-file which the compiler is currently reading/processing is not a good idea. Aug 10, 2020 at 9:44