Concurrently interleaving execution of Lua and TeX in LuaTeX

Question

The usual way to interleave Lua and TeX in generating documents is by invoking Lua from Tex through the \directlua facility and generate Tex from Lua using tex.print(). For example, consider the Plain LuaTeX:

\directlua{
    es=""; someluatex = function () tex.everypar="PAR!" end
    tex.print("one", es, "two")
    someluatex()
    tex.print("three", es, "four")
}
five\bye

Looking at the document LuaTeX generates for that input, we see that someluatex() is run before the Tex backend does anything with the output: the PAR!s will be inserted before "one" and "two", as well as after "three".

Is there best practice for ensuring that TeX code issued by some Lua code is issued before some other LuaTeX code? There are several ways to do this: with some code I am working with I am trying the following scheme:

1. Wrap the code executed by the \directlua in a coroutine, call it myco;

2. At the point we want to ensure the Tex so far tex.print-ed is executed, use:

   tex.print("\\directlua{coroutine.resume(myco)}")
   coroutine.yield()
   

3. If everything is in order, the Lua will resume after the Tex is processed.

This is not ideal, however: we must ensure that the TeX backend is in the right state to understand the \directlua command, which may not always be the case.

Is there a better way to achieve this?

Answer 1

Short answer: You have no other choice but using some kind of directlua/coroutine trick.

Longer answer: It doesn't work that way if I am correct. I can show a trick I have been using for quite some time (you mention the coroutine, so you might already use that, but for the record I'll write it down here). The problem is, as is pointed out in the question, that LuaTeX (the TeX side) does not execute the \directlua "command" until the closing brace }.

So, let's assume you are building a box of some kind and want to find out how big the box is. The simple approach does not work:

\directlua{
  tex.print("\\setbox20=\\hbox{foo}")
  print(tex.box[20].width)
}

because TeX has not typeset the box when accessing the width of the box. So what you need to do is

\directlua{
  tex.print("\\setbox20=\\hbox{foo}")
}

\directlua{
  print(tex.box[20].width)
}

This soon gets ugly, because the control flow is still on TeX's side. If you need more and more Lua code, you want to have something like a directtex("...") function from Lua. There is none, but you can us the mentioned trick with coroutines to do this. coroutines is a programming concept in Lua (and other languages) which lets you jump forth and back between two places in a program. So the idea is: jump into the Lua code, and go back when you need TeX and immediately go back to your Lua code where you were before.

The idea is to create a function main_loop() and create a coroutine with this argument. Then you create a coroutine with this function as the entry point: co = coroutine.create(main_loop) and jump into the coroutine (the main_loop() function) with coroutine.resume(co). Now when in Lua code, you can say coroutine.yield("\\some\\TeX\\code") and the control flow jumps back to the place where you called coroutine.resume(). In other words (document.tex):

\directlua{ dofile("luacode.lua") }

\newif\ifcontinue
\continuetrue

\directlua { co = coroutine.create(main_loop) }
\loop \directlua{  ok,b=coroutine.resume(co)  tex.sprint(b) }\ifcontinue  \repeat

\bye

and luacode.lua:

function directtex(str)
  coroutine.yield(str)
end

function main_loop()
  directtex("\\setbox20\\hbox{foo}")
  print(tex.box[20].width))
  directtex("\\continuefalse")
end

The heart of this is the TeX-loop around the coroutine.resume and tex.sprint(). To jump out of this infinite loop, I have this new boolean continue. Now you have the control flow on the Lua side of your program and you can choose to go back to TeX whenever you need to. Just end your program with directtex("\\continuefalse") and the loop will end.

There is a drawback however. Error handling is now really awful. If you have an error in your Lua code, the computer blows up or the world will end, but LuaTeX might keep on running. A solution to get a better error handling is to use the pcall() lua function:

function call(...)
  local ret = { pcall(...) }
  if ret[1]==false then
    texio.write_nl("Error:" .. tostring(ret[2]))
    directtex("\\continuefalse")
  end
  return unpack(ret,2)
end

call(func,arg1,arg2,...)

But this will not catch the case where you say: directtex("\\csname")

Answer 2

(As detailed in the comments, this is more of a comment than an answer.)

I think you're mistaking tex.everypar="..." for inserting \everypar{...} into the TeX input stream. Using tex.everypar="..." you are setting the \everypar token register at the point where you use \directlua, while it seems that what you actually want is to insert the setting into the TeX input stream

\directlua{
    es=""; someluatex = function () tex.print("\string\\everypar{PAR!}") end
    tex.print("one", es, "two")
    someluatex()
    tex.print("three", es, "four")
}
five\bye

My example means that TeX sees the tokens

one

two \everypar{PAR!} three

four five\bye

whereas your original is actually the same as

\everypar{PAR!}% Occurs where \directlua was
one

two three

four five\bye