# Concurrently interleaving execution of Lua and TeX in LuaTeX

The usual way to interleave Lua and TeX in generating documents is by invoking Lua from Tex through the \directlua facility and generate Tex from Lua using tex.print(). For example, consider the Plain LuaTeX:

\directlua{
es=""; someluatex = function () tex.everypar="PAR!" end
tex.print("one", es, "two")
someluatex()
tex.print("three", es, "four")
}
five\bye

Looking at the document LuaTeX generates for that input, we see that someluatex() is run before the Tex backend does anything with the output: the PAR!s will be inserted before "one" and "two", as well as after "three".

Is there best practice for ensuring that TeX code issued by some Lua code is issued before some other LuaTeX code? There are several ways to do this: with some code I am working with I am trying the following scheme:

1. Wrap the code executed by the \directlua in a coroutine, call it myco;
2. At the point we want to ensure the Tex so far tex.print-ed is executed, use:

tex.print("\\directlua{coroutine.resume(myco)}")
coroutine.yield()

3. If everything is in order, the Lua will resume after the Tex is processed.

This is not ideal, however: we must ensure that the TeX backend is in the right state to understand the \directlua command, which may not always be the case.

Is there a better way to achieve this?

-
Can you elaborate on the last comment that TeX is not always in a state to understand \directlua? – topskip Jun 16 '11 at 9:35
@Patrick: Maybe the previous command was tex.print("\csname "). – Charles Stewart Jun 16 '11 at 9:37
I guess that there is some real-world example where simply splitting up the Lua use into separate parts is not practical? From the LuaTeX docs, it's always seemed clear to me that TeX does nothing until \directlua finishes, so I don't follow the 'correct state' statement. – Joseph Wright Jun 16 '11 at 11:13
@Joseph: Right now, I'm playing with a read-eval-print routine, where Lua interacts with the user and feeds lines of Tex to the Tex engine, but I've thought about this problem before: any case where you want some Lua to use the Tex engine to build up some structure, then use Luatex functions to interrogate or mutate that structure and then issue further Tex code that depends on the results of executing that Luatex might benefit from this approach to code construction. – Charles Stewart Jun 16 '11 at 11:22

Short answer: You have no other choice but using some kind of directlua/coroutine trick.

Longer answer: It doesn't work that way if I am correct. I can show a trick I have been using for quite some time (you mention the coroutine, so you might already use that, but for the record I'll write it down here). The problem is, as is pointed out in the question, that LuaTeX (the TeX side) does not execute the \directlua "command" until the closing brace }.

So, let's assume you are building a box of some kind and want to find out how big the box is. The simple approach does not work:

\directlua{
tex.print("\\setbox20=\\hbox{foo}")
print(tex.box[20].width)
}

because TeX has not typeset the box when accessing the width of the box. So what you need to do is

\directlua{
tex.print("\\setbox20=\\hbox{foo}")
}

\directlua{
print(tex.box[20].width)
}

This soon gets ugly, because the control flow is still on TeX's side. If you need more and more Lua code, you want to have something like a directtex("...") function from Lua. There is none, but you can us the mentioned trick with coroutines to do this. coroutines is a programming concept in Lua (and other languages) which lets you jump forth and back between two places in a program. So the idea is: jump into the Lua code, and go back when you need TeX and immediately go back to your Lua code where you were before.

The idea is to create a function main_loop() and create a coroutine with this argument. Then you create a coroutine with this function as the entry point: co = coroutine.create(main_loop) and jump into the coroutine (the main_loop() function) with coroutine.resume(co). Now when in Lua code, you can say coroutine.yield("\\some\\TeX\\code") and the control flow jumps back to the place where you called coroutine.resume(). In other words (document.tex):

\directlua{ dofile("luacode.lua") }

\newif\ifcontinue
\continuetrue

\directlua { co = coroutine.create(main_loop) }
\loop \directlua{  ok,b=coroutine.resume(co)  tex.sprint(b) }\ifcontinue  \repeat

\bye

and luacode.lua:

function directtex(str)
coroutine.yield(str)
end

function main_loop()
directtex("\\setbox20\\hbox{foo}")
print(tex.box[20].width))
directtex("\\continuefalse")
end

The heart of this is the TeX-loop around the coroutine.resume and tex.sprint(). To jump out of this infinite loop, I have this new boolean continue. Now you have the control flow on the Lua side of your program and you can choose to go back to TeX whenever you need to. Just end your program with directtex("\\continuefalse") and the loop will end.

There is a drawback however. Error handling is now really awful. If you have an error in your Lua code, the computer blows up or the world will end, but LuaTeX might keep on running. A solution to get a better error handling is to use the pcall() lua function:

function call(...)
local ret = { pcall(...) }
if ret[1]==false then
texio.write_nl("Error:" .. tostring(ret[2]))
directtex("\\continuefalse")
end
return unpack(ret,2)
end

call(func,arg1,arg2,...)

But this will not catch the case where you say: directtex("\\csname")

-
Two observations: your code and my idea are pretty much the same, except the coroutine.resume is called within a Tex loop in your code, and inserted using tex.print in mine. I'm not sure which is more elegant. The point about error handling maybe suggests it is best to keep loops out of Tex where it is easy to do so. – Charles Stewart Jun 16 '11 at 14:18
Second: if the only way to have Lua invoke Tex is by putting Tex commands in the virtual buffer and then exit to have them expanded, then I agree that \directlua and coroutines must be the way to go. But if there is some other instrumentation of the Tex back end... And there may be other ideas about how to manage the Tex state problem. – Charles Stewart Jun 16 '11 at 14:21
@Charles: it seems as your wishes are being heard: tracker.luatex.org/view.php?id=682 – topskip Jun 16 '11 at 16:17
tex.execute(): That would be nice! Kudos for the link. – Charles Stewart Jun 16 '11 at 17:39

(As detailed in the comments, this is more of a comment than an answer.)

I think you're mistaking tex.everypar="..." for inserting \everypar{...} into the TeX input stream. Using tex.everypar="..." you are setting the \everypar token register at the point where you use \directlua, while it seems that what you actually want is to insert the setting into the TeX input stream

\directlua{
es=""; someluatex = function () tex.print("\string\\everypar{PAR!}") end
tex.print("one", es, "two")
someluatex()
tex.print("three", es, "four")
}
five\bye

My example means that TeX sees the tokens

one

two \everypar{PAR!} three

four five\bye

whereas your original is actually the same as

\everypar{PAR!}% Occurs where \directlua was
one

two three

four five\bye
-
I think what you describe is exactly Charles' problem. Another example is when you say \directlua{tex.print("\\setbox20=\\hbox{foo}"); print(tex.box[20].width)} you get undesired results. – topskip Jun 16 '11 at 10:51
Your diagnosis of the problem is right, but I don't want to translate Lua code into Tex. Patrick's example works better than mine for seeing why this might not be workable. – Charles Stewart Jun 16 '11 at 11:03
@Patrick: Perhaps you might copy your comment on the question, as I can then delete my answer (which it seems is not one). – Joseph Wright Jun 16 '11 at 11:20
@Joseph you can do with a single directlua call using coroutines as shown by Charles above, thats a pretty neat trick :). I do for example 99.9% of my things in Lua and only jump back to TeX when I really need to, so I have no other choice but using some kind of directlua/coroutine trick to get the calculations (as seen above) working. See also a google-translated entry in my blog: translate.google.com/… – topskip Jun 16 '11 at 11:20
@Patrick: Perhaps you might convert your comment to an answer: it seems that this is the 'best practice' approach. – Joseph Wright Jun 16 '11 at 11:40