3

(Started with a more complex issue, but debugging while writing this question narrowed it down to what's below.)

Consider a file luatexfilelimit.lua containing the following:

-- Create the file (once)
local filename = 'luatexfilelimit.txt'
local command = string.format('echo "hello world" > "%s"', filename)
os.execute(command)

-- Print its contents (many times)
for i = 1, 3000000 do
   print(string.format('Attempt %d', i))
   for line in io.lines(filename) do
      print(line)
   end
end

And a .tex file containing just:

\directlua{dofile('luatexfilelimit.lua')}

Now when I run this .tex file with luatex -shell-escape or lualatex -shell-escape (actually we can even drop the -shell-escape if the file has already been created, and we can even invoke texlua luatexfilelimit.lua or luatex luatexfilelimit.lua), then pretty soon I get this error:

Attempt 249
hello world
Attempt 250
hello world
Attempt 251
latexfilelimit.lua:9: attempt to call a nil value
stack traceback:
    latexfilelimit.lua:9: in main chunk
    [C]: in function 'dofile'
    [\directlua]:1: in main chunk.
l.1 \directlua{dofile('latexfilelimit.lua')}

? 

It appears that io.lines returns nil, even though the file exists. When I change the limit in my shell for number of open file descriptors, with ulimit -n 100, I correspondingly get:

Attempt 94
hello world
Attempt 95
latexfilelimit.lua:9: attempt to call a nil value
stack traceback:
    latexfilelimit.lua:9: in main chunk
    [C]: in function 'dofile'
    [\directlua]:1: in main chunk.
l.1 \directlua{dofile('latexfilelimit.lua')}

? 

But when I run the Lua file with a Lua interpreter (lua luatexfilelimit.lua) it manages the huge number of iterations just fine. To me this suggests that in LuaTeX, somehow io.lines does not close the file, as it's supposed to do, according to the documentation.

Is this correct? Is this a bug in LuaTeX?

(Tested on LuaTeX from TeX Live 2017, on macOS.)


Edit: Thanks to Henri Menke's answer for pointing out that the issue seems to be with the garbage collector in LuaTeX, and that adding collectgarbage() inside the loop (or even collectgarbage('step') a few times) seems to make the issue go away. What's surprising though is why the garbage-collector is even involved. See the following:

collectgarbage('stop')
for i = 1, 123456 do
   print(string.format('Step %d: memory %.2f KB', i, collectgarbage('count')))
   for line in io.lines('luatexfilelimit.txt') do print(line) end
end

Here the first line turns off the garbage-collector completely. Yet the above file, when run with lua, has no trouble going through the large number of iterations. This also matches my understanding of the documentation of io.lines, which says it automatically closes the file when it reaches the end (GC not needed). But when run with texlua or luatex, the above does not work. Also, consider this (bad) code, which opens files and does not close them:

collectgarbage('stop')
for i = 1, 123456 do
   local f = io.open('luatexfilelimit.txt')
   print(string.format('Step %d: memory %.2f KB', i, collectgarbage('count')))
   for line in io.lines('luatexfilelimit.txt') do print(line) end
end

When run with the Lua interpreter, the error message is actually helpful:

lua: luatexfilelimit.lua:5: cannot open file 'luatexfilelimit.txt' (Too many open files)
stack traceback:
    [C]: in function 'lines'
    luatexfilelimit.lua:5: in main chunk
    [C]: in ?

Whereas in LuaTeX, io.lines seems to silently return nil, with no additional information. So the issue in LuaTeX is something more than simply the garbage-collector being slow; I suspect LuaTeX has actually changed the behaviour of io.lines somehow so that it neither closes the file properly (relying on GC instead), nor throws an error properly.

  • Looks like a bug. – Henri Menke Dec 5 '17 at 3:43
  • Interestingly, texlua and texluajit fail at different numbers of open files which are both beyond the limit of open files on my system. It could be that the GC is too slow when embedded in LuaTeX. – Henri Menke Dec 5 '17 at 4:16
  • 1
    It's definitely not a problem of embedding in general because this minimal Lua interpreter does not exhibit the issue. – Henri Menke Dec 5 '17 at 4:38
  • Looks like adding five garbage-collection steps (for j = 1, 5 do collectgarbage('step') end) inside the loop works, but not with 4 in place of 5. Fun. – ShreevatsaR Dec 5 '17 at 6:33
  • 1
    I posted this issue to the ConTeXt mailing list: mailman.ntg.nl/pipermail/ntg-context/2017/090368.html – Henri Menke Dec 5 '17 at 21:48
2

I don't know why, but for some reason the file handles are not automatically garbage collected. If you run collectgarbage("collect") in the loop it works. Writing to a file via os.execute and echo is platform-dependent and inefficient. I replaced by a more canonical approach.

-- Create the file (once)
local filename = 'luatexfilelimit.txt'
local file = io.open(filename, "w")
file:write("hello world\n")
file:close()

-- Print its contents (many times)
for i = 1, 3000000 do
   print(string.format('Attempt %d', i))
   for line in io.lines(filename) do
      print(line)
   end
   collectgarbage("collect")
end

It also works with an explicit file handle (which is closed explicitly).

-- Create the file (once)
local filename = 'luatexfilelimit.txt'
local file = io.open(filename, "w")
file:write("hello world\n")
file:close()

-- Print its contents (many times)
for i = 1, 3000000 do
   print(string.format('Attempt %d', i))
   local file = io.open(filename, "r")
   for line in file:lines() do
      print(line)
   end
   file:close()
end
  • Nice, thanks… If it's a real bug (as it appears), it may be something to mention on the LuaTeX mailing list or bug tracker, but I'm not subscribed to it. Adding collectgarbage() works. What's surprising to me is that (as I understand it) io.lines is supposed to work like the second code sample above (with explicit file:close()) — some documentation says that when io.lines reaches the end of the file, it closes the file. So garbage-collection should not even be necessary as I understand it. – ShreevatsaR Dec 5 '17 at 5:17
  • Thanks for reporting it; it looks like this is fixed in LuaTeX upstream now. – ShreevatsaR Dec 8 '17 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.