Accessing to .log messages from LuaTeX. Is it possible?

Question

Reading the question How to visualize the underfull box in final output PDF files I thought that it could be possible to use LuaTeX to intercept the "Underfull vbox" message and draw some kind of overlayed mark (using tikz or whatever) at that point. Probably not very useful, but fun to try as a proof of concept.

Before dealing with the appropriate point at which introduce the mark, or the pattern expression required to capture the relevant info, I tried first a simpler goal, but I was unsuccessful.

My goal was, from lualatex, write a document which produces a copy of the .log file into another .lbk file (or any other extension, that's not relevant). I tried the following approaches:

1. I wrote a lua function copylog() which simply opens the .log, read all its contents, and write those back in other file:

   function copylog()
     local f = io.open(tex.jobname .. ".log", "r")
     local fo = io.open(tex.jobname .. ".lbk", "w") 
     s = f:read("*all")
     f:close()
     fo:write(s)
     fo:close() 
   end 
   

2. In the document I wrote \AtBeginDocument{\directlua{copylog()}} (I wanted to read the log from the previous compilation).

   This was partially successful. The resulting file contained part of the log, but not complete, it stopped a few lines before than the \begin{document} was processed.

   I realized that I was not reading the log from previous compilation, but instead the log of the current compilation. As soon as lualatex was run, a new log file was opened, a lot of contents from the preamble was written and then my lua code read all that stuff (but not what will come later).

3. So I changed the point at which my function was executed and wrote: \AtEndDocument{\directlua{copylog()}}, trying to process the log from the current compilation, when it was done.

   This failed again. This time I got a few lines more from the log file, but not the complete log file. Somehow I expected that, because at the point where my lua code reads the log it is still not complete (extra info and statistics are dumped even after the end of the document), but at least I expected it to contain part of the warning messages generated by my document. No luck, the copy ended again before \begin{document} was run.

   I realized that this was due to the buffered output. Luatex already wrote the information I wanted in the file, but this info was still in some file buffers, not actually written in the disk, so not accessible to my lua function.

4. Since \end{document} was still to early to hook my function, I tried also using the end_run callback, but got the same results. At end_run the log file is still open, so I cannot get its full contents (because of the buffered output again).

From these experiments I concluded two questions:

1. Is it possible to read the .log from the previous compilation before lualatex opens it (and cleans it) again? I guess the answer is no. Apparently the .log file is the first output which is open, even before the document is loaded.
2. It is possible to force, from lua or from tex, a flush() of the log file from lua, to be able to read all the messages generated up to \end{document}?

If the two question above have negative answer, then new questions arrive:

3. There is some callback I could use to get the log messages or terminal output as they are written? I tried the obvious ones: process_output_buffer and show_error_hook. The first one didn't get any result. The second one was only activated by error messages, not by warnings.
4. Any other idea? I only want to capture the warning messages. Should not be that difficult...

Any expert in luatex internals can help me?

Answer 1

From the comments received so far, and my own conclusions after reading the LuaTeX manual, I'm now convinced that the log file cannot be read reliably from lua code being run in the same document whose log file I'm trying to read.

Opening that file from lua and reading with file.read() is not reliable because the same file is open and being written in by TeX, and since file output is buffered, even if I read the file at the end of the run, I don't get its full contents.

The other approach, to use some callback to get the log messages while they are written in the log file, is unfeasible because LuaTeX does not provide any callback to process log messages. The closest thing is show_error_hook but this callback is only called at the error messages and not at the informative messages nor warning messages.

In short, the answer to the question posed in the title is NO.

Fortunately I was able to solve my original problem in a different, and IMHO better, way.