6

I have a project depending on LuaLaTeX where the main document first generates one or more intermediate .tex files and then invokes lualatex in a loop to generate the final PDF results. This can produce up to 10-15 documents in one go. Later it may be possible that this will scale to a (web) server and produce these 10-15 documents (concretely: a musical full score plus an arbitrary number of instrumental parts) for a large number of works so we may end up with a three-digit number of lualatex invocations (in theory, when the project is complete and all scores should ever be (re)generated at once it might even reach a four-digit number.

For efficiency and for study purposes I would like to implement a multithreaded job queue for that second stage of the process where all to-be-generated documents are added to a queue and processed by a given number of worker threads (where each one launches an external process and waits for that to finish). I basically know how that works but from what I found on the net I have the suspicion that Lua doesn't provide the necessary multithreading support to do that.

So the question: does Lua (and particularly in the LuaLaTeX context) provide the machinery to reliably implement such a job queue? If so I'd be glad about pointers. Alternatively I would factor out the second stage to a single invocation of a Python script where I will surely find my way - but I'd actually prefer staying within the domain of the main Lua .tex document.

  • 2
    Could you make up a small example document how your document setup currently looks like? Lua has at least basic support for multi-threading which should be sufficient (keyword coroutines). – TeXnician Jun 8 '18 at 16:40
  • @TeXnician No, Lua has no multi-threading support. Coroutines are a very different concept to threads. – Henri Menke Jun 9 '18 at 1:00
  • Instead of invoking LuaLaTeX directly you could invoke it through GNU Parallel which allows you to easily run multiply shell commands in parallel. – Henri Menke Jun 9 '18 at 1:01
  • @HenriMenke Yes, but it should be sufficient for the needs of this task as you do not need preemption as far as I can tell from the OP. – TeXnician Jun 9 '18 at 6:47
  • This (lua.org/pil/9.4.html) seems to indicate that indeed coroutines might be a solution to my task. However, I'm still not sure if I really understand the mechanics of them. What I need is one array (of file names), start a number of coroutines (?) that launch an external lualatex process (without blocking), and whenever one of these processes finishes the next filename from the array is passed to a new invocation of the coroutine. I have the impression that's what I can do with them, but I still have to figure out how to handle the blocking/nonblocking. – uli_1973 Jun 9 '18 at 7:10
7

Below you can find a poor man's task pool. It makes use of the fact that io.popen doesn't block. To wait for a job you can use :close() on its file handle. Below is a simple example where every job just sleeps five seconds and then creates a new empty file.

local scheduled = {}
local running = {}
local jobs = {
    new = function(cmd)
        local fn = function()
            local handle = io.popen(cmd)
            table.insert(running, handle)
        end
        table.insert(scheduled, fn)
    end,
    run = function(nprogs)
        while true do
            if #scheduled == 0 then -- no more jobs?
                -- close remaining jobs and exit
                for _,job in ipairs(running) do
                    job:close()
                end
                break
            end
            if #running == nprogs then -- maximum number of jobs running?
                -- block and wait for the first one
                running[1]:close()
                table.remove(running,1)
            end
            -- start new job
            scheduled[1]()
            table.remove(scheduled,1)
        end
    end
}

jobs.new("sleep 5; touch out1")
jobs.new("sleep 5; touch out2")
jobs.new("sleep 5; touch out3")
jobs.new("sleep 5; touch out4")
jobs.new("sleep 5; touch out5")
jobs.new("sleep 5; touch out6")
jobs.new("sleep 5; touch out7")
jobs.new("sleep 5; touch out8")
jobs.new("sleep 5; touch out9")
jobs.run(4)

If the jobs were executed serially it would take 45 s. I simply wait for the job on top of the stack to complete, rather than any job on the stack. Hence this only gives you a good speedup if all jobs take roughly the same time (or you schedule faster jobs before slower ones).

$ time lua test.lua 

real    0m15.015s
user    0m0.028s
sys 0m0.011s

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.