Implement a job queue in LuaLaTeX

Question

I have a project depending on LuaLaTeX where the main document first generates one or more intermediate .tex files and then invokes lualatex in a loop to generate the final PDF results. This can produce up to 10-15 documents in one go. Later it may be possible that this will scale to a (web) server and produce these 10-15 documents (concretely: a musical full score plus an arbitrary number of instrumental parts) for a large number of works so we may end up with a three-digit number of lualatex invocations (in theory, when the project is complete and all scores should ever be (re)generated at once it might even reach a four-digit number.

For efficiency and for study purposes I would like to implement a multithreaded job queue for that second stage of the process where all to-be-generated documents are added to a queue and processed by a given number of worker threads (where each one launches an external process and waits for that to finish). I basically know how that works but from what I found on the net I have the suspicion that Lua doesn't provide the necessary multithreading support to do that.

So the question: does Lua (and particularly in the LuaLaTeX context) provide the machinery to reliably implement such a job queue? If so I'd be glad about pointers. Alternatively I would factor out the second stage to a single invocation of a Python script where I will surely find my way - but I'd actually prefer staying within the domain of the main Lua .tex document.

Answer 1

Below you can find a poor man's task pool. It makes use of the fact that io.popen doesn't block. To wait for a job you can use :close() on its file handle. Below is a simple example where every job just sleeps five seconds and then creates a new empty file.

local scheduled = {}
local running = {}
local jobs = {
    new = function(cmd)
        local fn = function()
            local handle = io.popen(cmd)
            table.insert(running, handle)
        end
        table.insert(scheduled, fn)
    end,
    run = function(nprogs)
        while true do
            if #scheduled == 0 then -- no more jobs?
                -- close remaining jobs and exit
                for _,job in ipairs(running) do
                    job:close()
                end
                break
            end
            if #running == nprogs then -- maximum number of jobs running?
                -- block and wait for the first one
                running[1]:close()
                table.remove(running,1)
            end
            -- start new job
            scheduled[1]()
            table.remove(scheduled,1)
        end
    end
}

jobs.new("sleep 5; touch out1")
jobs.new("sleep 5; touch out2")
jobs.new("sleep 5; touch out3")
jobs.new("sleep 5; touch out4")
jobs.new("sleep 5; touch out5")
jobs.new("sleep 5; touch out6")
jobs.new("sleep 5; touch out7")
jobs.new("sleep 5; touch out8")
jobs.new("sleep 5; touch out9")
jobs.run(4)

If the jobs were executed serially it would take 45 s. I simply wait for the job on top of the stack to complete, rather than any job on the stack. Hence this only gives you a good speedup if all jobs take roughly the same time (or you schedule faster jobs before slower ones).

$ time lua test.lua 

real    0m15.015s
user    0m0.028s
sys 0m0.011s