How to rewrite tex file using lua

Question

I try to solve the problem with $ and $$ in formulas, to perform automatic tagging of these formulas, including adding in the structure tree. Variant with \grabinline or \grabdisplay does not work for me because I get an error when tryingto use equation, for example. Recently I read that I can work with my own files via lualatex. First I tried to simply replace $$ and $ if there is no \ character before, but not only does it not work, but my file is corrupted when I try to rewrite it, I mean, it adds new strange lines.

Please help me, how can I fix this problem, I mean the problem with the corruption of thefile, and please explain to me why it happens?

test.tex

\documentclass{article}
\directlua{require("test.lua")}
\pagestyle{empty}
\begin{document}
test
$$a+b=a^b$$ $a-b=a/b$
\end{document}

test.lua

f=io.open(tex.jobname..".tex","r+")
lines=f:lines()
for line in lines do
line=line:gsub("([^\\]?)$$(.+-)$$","%1\\[%2\\]")
line=line:gsub("([^\\]?)$(.+-)$","%1\\(%2\\)")
texio.write_nl("line of file "..line)
f:write(line)
end
f:close()

Answer 1

The problem is not TeX related, it is a general problem with reading and writing the same file. So what is going on?

The output you probably get is

\documentclass{article}
\documentclass{article}t.lua")}
t.lua")}le{empty}
le{empty}cument}
cument}a+b=a^b$$ $a-b=a/b$
a+b=a^b$$ $a-b=a/b$

This happens because after opening your original file, you are at the first byte of the first line. Then using lines, you read the first line: \documentclass{article}. After that, your position in the line is the beginning of the second line. Here you issue f:write, so the first, unmodified, line is written to the current position in the file, which is the second line. It overwrites existing content.

So now, the file contains:

\documentclass{article}
\documentclass{article}t.lua")}
\pagestyle{empty}
\begin{document}
test
$$a+b=a^b$$ $a-b=a/b$
\end{document}

and your current position is at the end of the second \documentclass{article}. Now your next lines iteration reads the rest of the line, so you get t.lua")}. Then you are at the start of the third line and you overwrite it with the just read text, so you get

documentclass{article}
\documentclass{article}t.lua")}
t.lua")}le{empty}
\begin{document}
test
$$a+b=a^b$$ $a-b=a/b$
\end{document}

This repeats for every line, until you get the file you observed.

Which lesson can you learn here: Do not overwrite a file while you are still reading from it!

In contrast some other programming languages contain functions, like Python with readlines, which on the first glance look similar to Lua's io.lines. But e.g. Python's readlines directly reads all lines into an array, while Lua only reads one line at a time in every iteration of the loop. So no quite so obvious problems appear with similar code there because reading and writing is clearly separated: Reading only during readline, writing afterwards.

Anyway rewriting your TeX source file during the TeX run is very dangerous and also not compatible (Windows is sensible when it comes to editing open files).

It is much safer and easier to use process_input_buffer if you only want to change one line at a time. You can use status.input_ptr to only affect lines from specific files: (I fixed the pattern in the process)

luatexbase.add_to_callback("process_input_buffer", function(line)
  if status.input_ptr ~= 1 then return end -- Only change lines of the main file
  texio.write_nl("line of file "..line)
  print(line:match("([^\\]?)$$(.-)$%$"))
  return line:gsub("([^\\]?)$$(.-)$%$","%1\\[%2\\]")
             :gsub("([^\\]?)$(.-)%$","%1\\(%2\\)")
end, "my_math_rewrite")

Even if this is much less dangerous, I would still recommend looking for ways to archive your goals without rewriting input lines.

Answer 2

(a little bit late to the party, but hopefully still useful)

Here's a solution which doesn't create a new text file. Instead, it works "on the fly", essentially like a pre-processor, and replaces matching pairs of $$ and $, respectively, with their LaTeX-preferred expressions: This happens before TeX starts its usual work of expanding macros, etc.

Creating a new output text file, to write out the on-the-fly-modified contents of the input put, is left as a separate exercise.

To take this from the proof-of-concept stage to something that might be sufficiently robust for real-world work, one would certainly also have to add code to test whether the material being examined by the preprocessor happens to in verbatim mode or is contained in a URL string. In these and similar cases, no substitutions should be performed, right?

% !TEX TS-program = lualatex

%% Create external file to store Lua code:
\RequirePackage{filecontents}
\begin{filecontents*}{test.lua}

function substitute_dollar_symbols ( line )
    line = line:gsub ( "([^\\]?)$$(.+-)$$" , "%1\\[%2\\]" )
    line = line:gsub ( "([^\\]?)$(.+-)$"   , "%1\\(%2\\)" )
    return line
end

\end{filecontents*}

\documentclass{article}
%% Load Lua code from external file:
\directlua{require("test.lua")}

%% Create two LaTeX utility macros to (a) assign Lua 
%% function to the "process_input_buffer" callback 
%% and (b) remove the function from that callback:
\newcommand\SubstituteDollarSymbolsOn{%
  \directlua{luatexbase.add_to_callback (
    "process_input_buffer", 
    substitute_dollar_symbols , 
    "substitute_dollar_symbols" )}}
\newcommand\SubstituteDollarSymbolsOff{%
  \directlua{luatexbase.remove_from_callback (
    "process_input_buffer",
    "substitute_dollar_symbols" )}}

%% Activate the Lua function at start of document:
\AtBeginDocument{\SubstituteDollarSymbolsOn}

\begin{document}
test
$$a+b=a^b$$ $a-b=a/b$ % $$ $
abc $$uvw$$
\end{document}