# How to rewrite tex file using lua

I try to solve the problem with $ and $$ in formulas, to perform automatic tagging of these formulas, including adding in the structure tree. Variant with \grabinline or \grabdisplay does not work for me because I get an error when tryingto use equation, for example. Recently I read that I can work with my own files via lualatex. First I tried to simply replace $$ and $ if there is no \ character before, but not only does it not work, but my file is corrupted when I try to rewrite it, I mean, it adds new strange lines.

Please help me, how can I fix this problem, I mean the problem with the corruption of thefile, and please explain to me why it happens?

test.tex

\documentclass{article}
\directlua{require("test.lua")}
\pagestyle{empty}
\begin{document}
test
$$a+b=a^b$$ $a-b=a/b$
\end{document}


test.lua

f=io.open(tex.jobname..".tex","r+")
lines=f:lines()
for line in lines do
line=line:gsub("([^\\]?)$$(.+-)$$","%1\$%2\$")
line=line:gsub("([^\\]?)$(.+-)$","%1\$$%2\$$")
texio.write_nl("line of file "..line)
f:write(line)
end
f:close()

• Why are you opening the input file a second time but in Lua? Why are writing the substituted content to the log instead of a new file? – Henri Menke Aug 12 at 2:59
• Please provide more background as to what you're trying to achieve. Separately, would you be ok with a LuaLaTeX-based solution that performs the substitution of the $$ and  symbols "on the fly", i.e., without writing its modified input stream back to file? – Mico Aug 12 at 4:22 • Maybe a solution outside of LaTeX would be easier, i.e., preprocess the file with any programming language or command line tool that you prefer and run LaTeX afterwards? – Marijn Aug 12 at 10:05 • @Henri Menke Because i want to do modification of my file,and compile after that modifiered file,but help me please,how i can rewrite existing file in lua? Why in lualatex problem with this? I even tryed open existing file in read mode,get lines,close it and open it in w mode,but in this case i can't works with line variable. Explain me please,why my file corrupted and how rewrite existing file. I not ask you now help me with substitutions,i only help to you help me with rewriting an existing file. Thanks everybody for the help. – Aleksandr Kozlovskiy Aug 12 at 10:24 • @Mico i am not understood your idea,how i can do replacing  and$$ without modifying of file? And it very interesting for me,what the secret of lualatex,that it corrupted the file and write strange lines. Why my code with rewriting of file not works. Can someone please explain to me this secret of lualatex. I dont want use special tools for substitution,because i want to implement this solution in my package and if i will do it in lualatex,i can do my solution more crossplatform.I hope that you will explain to me your idea solution of my problem. – Aleksandr Kozlovskiy Aug 12 at 16:55

The problem is not TeX related, it is a general problem with reading and writing the same file. So what is going on?

The output you probably get is

\documentclass{article}
\documentclass{article}t.lua")}
t.lua")}le{empty}
le{empty}cument}
cument}a+b=a^b$$a-b=a/b a+b=a^b$$ $a-b=a/b$


This happens because after opening your original file, you are at the first byte of the first line. Then using lines, you read the first line: \documentclass{article}. After that, your position in the line is the beginning of the second line. Here you issue f:write, so the first, unmodified, line is written to the current position in the file, which is the second line. It overwrites existing content.

So now, the file contains:

\documentclass{article}
\documentclass{article}t.lua")}
\pagestyle{empty}
\begin{document}
test
$$a+b=a^b$$ $a-b=a/b$
\end{document}


and your current position is at the end of the second \documentclass{article}. Now your next lines iteration reads the rest of the line, so you get t.lua")}. Then you are at the start of the third line and you overwrite it with the just read text, so you get

documentclass{article}
\documentclass{article}t.lua")}
t.lua")}le{empty}
\begin{document}
test
$$a+b=a^b$$ $a-b=a/b$
\end{document}


This repeats for every line, until you get the file you observed.

Which lesson can you learn here: Do not overwrite a file while you are still reading from it!

In contrast some other programming languages contain functions, like Python with readlines, which on the first glance look similar to Lua's io.lines. But e.g. Python's readlines directly reads all lines into an array, while Lua only reads one line at a time in every iteration of the loop. So no quite so obvious problems appear with similar code there because reading and writing is clearly separated: Reading only during readline, writing afterwards.

Anyway rewriting your TeX source file during the TeX run is very dangerous and also not compatible (Windows is sensible when it comes to editing open files).

It is much safer and easier to use process_input_buffer if you only want to change one line at a time. You can use status.input_ptr to only affect lines from specific files: (I fixed the pattern in the process)

luatexbase.add_to_callback("process_input_buffer", function(line)
if status.input_ptr ~= 1 then return end -- Only change lines of the main file
texio.write_nl("line of file "..line)
print(line:match("([^\\]?)$$(.-)%")) return line:gsub("([^\\]?)$$(.-)$%$","%1\$%2\$")
:gsub("([^\\]?)$(.-)%$","%1\$$%2\$$")
end, "my_math_rewrite")


Even if this is much less dangerous, I would still recommend looking for ways to archive your goals without rewriting input lines.

• %c3%bcger Marcel Krüger yes,my file have similar structure,and i want to understood,it issue of lua,or it normal works of lua,because in python i can without any problem rewrite the same file. and for lua,as i understood,i must add all lines of file in table,than open it in write mode and after that i can write again my table in file. but it seems,when i get lines of file,i loose the last character of each line,so i dont know,if line finish by \n or \r character. May be it exists some ways to solve this problems? – Aleksandr Kozlovskiy Aug 12 at 22:28
• I desided to use for substitutions modifying of file,because at first i can get emmidiately all lines of file and at second in process_input_buffer i get not only lines of documents,but also line of all packages,which i use in document and also some latex lines. – Aleksandr Kozlovskiy Aug 12 at 22:29
• @AleksandrKozlovskiy Are you using readlines in your Python code? – Marcel Krüger Aug 12 at 22:32
abc $$uvw$$
\end{document}