I have a project depending on LuaLaTeX where the main document first generates one or more intermediate
.tex files and then invokes
lualatex in a loop to generate the final PDF results. This can produce up to 10-15 documents in one go. Later it may be possible that this will scale to a (web) server and produce these 10-15 documents (concretely: a musical full score plus an arbitrary number of instrumental parts) for a large number of works so we may end up with a three-digit number of
lualatex invocations (in theory, when the project is complete and all scores should ever be (re)generated at once it might even reach a four-digit number.
For efficiency and for study purposes I would like to implement a multithreaded job queue for that second stage of the process where all to-be-generated documents are added to a queue and processed by a given number of worker threads (where each one launches an external process and waits for that to finish). I basically know how that works but from what I found on the net I have the suspicion that Lua doesn't provide the necessary multithreading support to do that.
So the question: does Lua (and particularly in the LuaLaTeX context) provide the machinery to reliably implement such a job queue? If so I'd be glad about pointers.
Alternatively I would factor out the second stage to a single invocation of a Python script where I will surely find my way - but I'd actually prefer staying within the domain of the main Lua
lualatexprocess (without blocking), and whenever one of these processes finishes the next filename from the array is passed to a new invocation of the coroutine. I have the impression that's what I can do with them, but I still have to figure out how to handle the blocking/nonblocking.