I have a project depending on LuaLaTeX where the main document first generates one or more intermediate
.tex files and then invokes
lualatex in a loop to generate the final PDF results. This can produce up to 10-15 documents in one go. Later it may be possible that this will scale to a (web) server and produce these 10-15 documents (concretely: a musical full score plus an arbitrary number of instrumental parts) for a large number of works so we may end up with a three-digit number of
lualatex invocations (in theory, when the project is complete and all scores should ever be (re)generated at once it might even reach a four-digit number.
For efficiency and for study purposes I would like to implement a multithreaded job queue for that second stage of the process where all to-be-generated documents are added to a queue and processed by a given number of worker threads (where each one launches an external process and waits for that to finish). I basically know how that works but from what I found on the net I have the suspicion that Lua doesn't provide the necessary multithreading support to do that.
So the question: does Lua (and particularly in the LuaLaTeX context) provide the machinery to reliably implement such a job queue? If so I'd be glad about pointers.
Alternatively I would factor out the second stage to a single invocation of a Python script where I will surely find my way - but I'd actually prefer staying within the domain of the main Lua