I have a project depending on LuaLaTeX where the main document first generates one or more intermediate .tex files and then invokes lualatex in a loop to generate the final PDF results. This can produce up to 10-15 documents in one go. Later it may be possible that this will scale to a (web) server and produce these 10-15 documents (concretely: a musical full score plus an arbitrary number of instrumental parts) for a large number of works so we may end up with a three-digit number of lualatex invocations (in theory, when the project is complete and all scores should ever be (re)generated at once it might even reach a four-digit number.

For efficiency and for study purposes I would like to implement a multithreaded job queue for that second stage of the process where all to-be-generated documents are added to a queue and processed by a given number of worker threads (where each one launches an external process and waits for that to finish). I basically know how that works but from what I found on the net I have the suspicion that Lua doesn't provide the necessary multithreading support to do that.

So the question: does Lua (and particularly in the LuaLaTeX context) provide the machinery to reliably implement such a job queue? If so I'd be glad about pointers. Alternatively I would factor out the second stage to a single invocation of a Python script where I will surely find my way - but I'd actually prefer staying within the domain of the main Lua .tex document.

  • 2
    Could you make up a small example document how your document setup currently looks like? Lua has at least basic support for multi-threading which should be sufficient (keyword coroutines).
    – TeXnician
    Jun 8, 2018 at 16:40
  • @TeXnician No, Lua has no multi-threading support. Coroutines are a very different concept to threads. Jun 9, 2018 at 1:00
  • Instead of invoking LuaLaTeX directly you could invoke it through GNU Parallel which allows you to easily run multiply shell commands in parallel. Jun 9, 2018 at 1:01
  • @HenriMenke Yes, but it should be sufficient for the needs of this task as you do not need preemption as far as I can tell from the OP.
    – TeXnician
    Jun 9, 2018 at 6:47
  • This (lua.org/pil/9.4.html) seems to indicate that indeed coroutines might be a solution to my task. However, I'm still not sure if I really understand the mechanics of them. What I need is one array (of file names), start a number of coroutines (?) that launch an external lualatex process (without blocking), and whenever one of these processes finishes the next filename from the array is passed to a new invocation of the coroutine. I have the impression that's what I can do with them, but I still have to figure out how to handle the blocking/nonblocking.
    – uli_1973
    Jun 9, 2018 at 7:10

1 Answer 1


Below you can find a poor man's task pool. It makes use of the fact that io.popen doesn't block. To wait for a job you can use :close() on its file handle. Below is a simple example where every job just sleeps five seconds and then creates a new empty file.

local scheduled = {}
local running = {}
local jobs = {
    new = function(cmd)
        local fn = function()
            local handle = io.popen(cmd)
            table.insert(running, handle)
        table.insert(scheduled, fn)
    run = function(nprogs)
        while true do
            if #scheduled == 0 then -- no more jobs?
                -- close remaining jobs and exit
                for _,job in ipairs(running) do
            if #running == nprogs then -- maximum number of jobs running?
                -- block and wait for the first one
            -- start new job

jobs.new("sleep 5; touch out1")
jobs.new("sleep 5; touch out2")
jobs.new("sleep 5; touch out3")
jobs.new("sleep 5; touch out4")
jobs.new("sleep 5; touch out5")
jobs.new("sleep 5; touch out6")
jobs.new("sleep 5; touch out7")
jobs.new("sleep 5; touch out8")
jobs.new("sleep 5; touch out9")

If the jobs were executed serially it would take 45 s. I simply wait for the job on top of the stack to complete, rather than any job on the stack. Hence this only gives you a good speedup if all jobs take roughly the same time (or you schedule faster jobs before slower ones).

$ time lua test.lua 

real    0m15.015s
user    0m0.028s
sys 0m0.011s

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .