# What are the advantages of Lua over other scripting languages for a TeX implementation?

The question is inspired by Martin Schröder's answer to I am new to TeX. Should I use LaTeX, XeLaTeX, …? as well as my two previous questions on this forum:

In retrospect, I am truly surprised that no LuaTeX user offered any answers to my questions as it seems to me that that could be easily done in LuaTeX by embedding Lua scripts into the LaTeX code. Can it be really done (question for LuaTeX users only) and is it easy?

My second question is motivated by the fact that I hate very thought of having to learn another scripting/general programming language (Lua) if I already know Python (you can substitute Perl if you like) just to use it with TeX. Is anybody cooking a Python interface for TeX call it PythonTeX for short (PerlTeX actually does exist)? As far as I know PythonTeX doesn't exist. There is something called PyTeX but that just look like a tool for people who want to type TeX documents using Python syntax.

Is there an advantage of using Lua over Python as an embedded scripting language for TeX (apart of the fact that people who started LuaTeX probably were more familiar with Lua)?

• With regard to your question on truth tables, see the example at the ConTeXt wiki. It is relatively straightforward to modify that to generate truth tables. I'll post a separate answer to your question on automatic document update. – Aditya Jan 29 '12 at 19:38
• Just a minor comment on the Lua bit. I had never even seen a bit of Lua code before about three months ago. Now I can code stuff like this: youtu.be/m_9Q2X-PJwE It really is easy to learn, even for a non-programmer like myself. – Loop Space Jan 29 '12 at 21:22
• @Andrew Stacey I will re-paraphrase one of my favorite comments on this forum which happened to be written by you. Some n00b was asking about perfect tools for TeX-ing to which you responded:"Do not forget to do your research while searching for the perfect tool". What I am trying to say? The human life span is very short and we have to budget carefully. I would much rather waist the rest of what I have left trying to do research than trying to learn new tools. Having said that I already broke that rule by learning Python while being fairly familiar with Shell, AWK, MATLAB and C. – Predrag Punosevac Jan 29 '12 at 21:32
• Touché. But I can turn that around. Learning Lua will take very little time at all. Writing PythonTeX will take an extremely long time. If going beyond TeX will help you do your day job, then learning Lua will leave much more time for the research than writing PythonTeX would. – Loop Space Jan 29 '12 at 21:57
• @Sharpie: plasmas cutters are expensive ;-) – Bruno Le Floch Jan 30 '12 at 7:05

The LuaTeX developers have commented on their choice of Lua over other languages, including Python, on their home page.

Embedding the interpreter is one thing, and apparently no fun with Python. Another is to actually make the innards of TeX visible to the embedded interpreter. While much of the communication code could likely be adapted in some way from LuaTeX to PythonTeX, it will still be a lot of work.

A better approach may be to cheat a bit, and simply piggyback on the LuaTeX interpreter. There is a bridge that 'embeds' Python inside Lua called Lunatic Python, and maybe it can be made to work with LuaTeX also. However, it may be easier to just use XML-RPC: Let LuaTeX spawn a Python process, which acts as the XML-RPC server. LuaTeX commands can then make calls to the Python process. Bonus feature: Once this is implemented on the LuaTeX side, it can be used with any other scripting language, not just Python.

• In EuroTeX 2009 Luigi Scarso demostrated a working version of Lunatic Python. More details on installation instruction along with different examples are on the ConTeXt wiki – Aditya Jan 29 '12 at 20:18
• Thanks, Aditya. I looked at it, and while there seems to be a bit of source code hacking involved, it doesn't seem too bad. Worth a try for anyone interested in exploring the possibilities. – Michael Palmer Jan 30 '12 at 2:36

I have been coding "PythonTeX" since last May, and am planning the first public release sometime between next weekend and the beginning of March. I'm actually planning to call it PythonTeX, by analogy to PerlTeX and SageTeX.

I've created a LaTeX package, and accompanying Python scripts, that provide most functionality of python.sty, SageTeX, and SympyTeX, but with emphasis on speed and usability. Here's a brief summary of what PythonTeX will bring.

• Persistence between commands and environments. For example, each environment \begin{pythoncode} ... \end{pythoncode} picks up where the last left off, so all variables, functions, etc. persist. You can optionally name commands and environments, and then there is only persistence between commands and environments with the same name. For example, \begin{pythoncode}[name] ... \end{pythoncode} only shares persistence with other commands and environments called name.

• Automatic inclusion of printed content. For example,

\begin{pythoncode}
print(1+1)
\end{pythoncode}


is automatically replaced by 2 in the compiled document.

• Speed. Each set of named commands and environments is executed in its own process, using Python's multiprocessing package. Also, each set is hashed, so code is only executed when changed. The results of running the code are saved, and reused when no changes are detected.

• Code typesetting. All code that is executed can also be typeset, using fancyvrb and its internals. Optionally, code can be typeset using Pygments, which is a Python package that does syntax highlighting (Pygments is used by the minted package). When Pygments is used, all code is hashed and the typeset results are saved, so that only changed code must be processed by Pygments in subsequent runs. This helps prevent Pygments from slowing things down.

• Meaningful error messages. All error messages are parsed by code that determines the line of your document where the error occurred (as opposed to the line of the *.py file that is actually executed). So you know exactly where things went wrong in your document.

• Minimal files. The code cleans up after itself, so the number of permanent files created by Python is kept to a minimum and all temporary files are deleted after each run.

I can post a link as soon as PythonTeX is released. I will also be submitting it to CTAN shortly after the first public release.

• Your PythonTeX is for LaTeX only or for plain TeX too? – Sam Qasbah Feb 1 '12 at 9:36
• You can't compare solutions like these (also perlTeX etc.) with the current state of LuaTeX, because the external processes will not allow to manipulate TeX's internal state and data structures. – topskip Feb 1 '12 at 9:39
• @SamQasbah Unfortunately, my package is for LaTeX, not plain TeX. It requires several other LaTeX packages like fancyvrb, so a plain TeX version could involve a lot of work. – G. Poore Feb 1 '12 at 16:18
• @PatrickGundlach I agree--PerlTeX, SageTeX, my "PythonTeX", and similar packages lack LuaTeX's power and integration on the TeX side. If someone needs that, then LuaTeX is the way to go (and I think Lua was a good language choice, given LuaTeX's objectives). But if someone just needs to run code and get its output back in a document, then these other packages can be compared with LuaTeX on that level. Personally, I don't need to manipulate TeX internals. I just want to run Python code that creates pdf figures or returns its output in LaTeX format, without having to leave my text editor. – G. Poore Feb 1 '12 at 17:05
• After some delays and feature creep, a beta version of my PythonTeX code is now available at GitHub. – G. Poore Apr 27 '12 at 19:12

My take is: Lua as a language was created as an embedded language and is well suited for the job. Now that with LuaTeX a scriptable TeX implementation exists, there is little principled reason to create another one. Of course, you can go ahead and create PythonTeX. But it seems that the community feels that this would not add enough value over LuaTeX to justify the cost.

• I do think it would add value (for me). – Neil G Jul 31 '12 at 9:26

Here is a way to call into a persistent Python process from LuaTex via JSON-RPC. 1)

Required setup:

2. Download a Lua json-rpc client. I adapted some Lua modules I found on the web and zipped them up and put them on my server. Unzip into one of the directories specified in this answer: "Lua tree" (analogue of texmf tree)

Now, we can write a Python server that exposes some functions:

import sys
from jsonrpclib.SimpleJSONRPCServer import SimpleJSONRPCServer

server = SimpleJSONRPCServer(('localhost', 8080))

server.register_function(lambda: "Hello from Python!", 'ping')

# the counter generator shows that the server can be used
# to preserve state between requests, and even across multiple
# LaTeX compiler runs (which we may not want)
counter = iter(xrange(sys.maxint))
server.register_function(lambda: counter.next(), 'count')

server.serve_forever()


Save this as testserver.py and run.

Here is test document that shows how to call the server:

\documentclass{article}

\directlua{ % make the running python server available as a proxy
require "jsonrpc"
pythonserver = jsonrpc.proxy("http://localhost:8080")
}

\newcommand{\python}[2][]{\directlua{%
result = pythonserver.#2(#1)
tex.print(result)
}}

\begin{document}

\python{ping}

The sum of 1,2,3,4,5,6,2,4,5,6 is \python[1,2,3,4,5,6,2,4,5,6]{add}

\directlua{%
for x=0,100,1 do
result = pythonserver.count()
tex.print(result)
end
}

\end{document}


Addressing the original question about how to implement pythontex, all that would be required now is a shell script that first starts the server, then calls luatex, and then shuts down the server again. This is left as an exercise to the reader ;)

1) The reason why I used JSON-RPC instead of XML-RPC is that I couldn't find a Lua-only XML-RPC client; the one client that I did find used a C extension library, and I and couldn't figure out how to load that into luatex (blush). The JSON-RPC modules were simpler, and after some kicking and screaming (on my part) could be persuaded to cooperate. Maybe someday I can figure out how to make XML-RPC work also.

Actually, a good way to use Python code into (plain) TeX code is:

1. for the typesetting work (boxes, glue and penalties) use only TeX;
2. for calculation, databases interface and so on you can build Python modules that you can call from the TeX program via \write18. The Python modules have to leave the output into files that have to be read via the usual \newread and then used into the TeX code. This solution work best on Linux, as far as on Linux Python is natively installed.

As an example try this: prepare the module adder.py' this way:

#!/usr/bin/env python

import sys

def save(data):
tempFile = open("temp.dat","w")
tempFile.write(str(data))
tempFile.close()

def double(x):
result = x * 2
save(result)

if __name__ == "__main__":
n = int(sys.argv[1])
double(n)
sys.exit


now, in the same directory, you can type a TeX file like this:

% file doubler.tex'
%
\def\double#1{%
\immediate\openin\tempfile=temp.dat
\var}%
%
Double 2 is \double2\par
Double 3 is \double3\par
Double 4 is \double4\par
Double 5 is \double5\par
Double 6 is \double6\par
\bye


This TeX file has to be compiled with shell-escape, e.g.:

tex -shell-escape doubler

pdftex -shell-escape doubler

xetex -shell-escape doubler

• I really appreciate your answer. I will have to investigate that a bit further. – Predrag Punosevac Jan 29 '12 at 21:41
• @PedragPunosevac I've added a working example into my post, so you can use that technique a bit easylier. – Sam Qasbah Jan 29 '12 at 22:55
• Why not simply \input the generated file? – Aditya Jan 29 '12 at 23:34
• This is nice, as far as toy examples are concerned. However, realize that this method creates and destroys a Python process for every call. On my machine, that is an overhead of 20 ms just for the Python interpreter itself. Load some big libraries such as scipy or matplotlib and it takes 120 ms, even with OS caching. So, for any serious usage, we need to talk to a persistent Python process, and the easiest way for this is XML-RPC. I just confirmed that with an asyncore-based XML-RPC server, a single call takes less than 2 ms. – Michael Palmer Jan 30 '12 at 2:29
• @MichaelPalmer could you post an example? Perhaps on this question, or I could ask another question "How do I interface between TeX and a persistent Python process?" (although... it is borderline off-topic for this site). – Bruno Le Floch Jan 30 '12 at 7:08

I think that the pdfTeX community analyzed the situation correctly when they chose Lua. Lua is intended to be embedded into other applications, and it provides a robust, easy to use C API. The Lua API is straightforward and its design eliminates the need for manual reference management in C code, unlike Python's API.

Python is an opinionated language controlled by essentially one person and many people, myself included do not favour the significance of whitespace.

Yes, for you it maybe easier to use Python as it would have been easier for me if everyone stayed with the original Pascal code. Pascal would have provided everything one needed, it is a beautiful language and everyone is almost using it daily via pseudo-code and it would have provided a good way to migrate TeX to a new generation.

If there is any merit to suggest another language, I would recommend JavaScript. It is gaining in popularity via the web and easy to learn. The maths part of it has already been written, at least for browsers. It also provides full unicode support.

• @ChristianLindig Roberto and his team come out more like the LaTeX Team with Roberto Ierusalimschy more like a guiding figure as well as an active developer. – Yiannis Lazarides Jan 29 '12 at 20:10
• Lua is intended to be embedded into other applications''-- Do you really think that there is a field in which python is worse than Lua, or viceversa? – Sam Qasbah Jan 29 '12 at 20:59
• @SamQasbah: Edited: It were the pdfTeX developers. – Martin Schröder Jan 29 '12 at 21:20
• I suggested JavaScript at the time, but any JS interpreter is a behemoth compared to the Lua interpreter. Lua is the better choice. – Martin Schröder Jan 29 '12 at 21:21
• @SamQasbah For answers I will let the current developers talk please see luatex.org/languages.html and yes I think there are fields where Lua is a better choice than python and vice versa. If you feel so strongly that this is the wrong language why don't you give it a try, find a group and produce pythonTeX? – Yiannis Lazarides Jan 29 '12 at 21:25

## Pweave

Much like the original poster, I also do regret the absence of a LaTeX interpreter capable of directly interpreting Python code.

Since PythonTeX got mentioned here in one of the other answers, I also would like to call your attention to Pweave. It is easier to use than PythonTex and also offers a couple more environment options.

Compiling a document that uses PythonTeX involves three steps: run latex, run pythontex.py, and ﬁnally run latex again. Pweave only needs two steps: run Pweave, then run pdflatex.

I got started with Pweave by reading a concise but excellent introductory article. You will also find my minimal working example over here on tex.StackExchange.

I have also posted an easier example involving currency notations and demonstrating the power of locales, Python and Pweave over here.

EDIT:

\$ Pweave -l


Pweave supported output formats:

• sphinx: reStructuredText for Sphinx
• pandoc: Pandoc markdown
• tex: Latex with verbatim for code and results
• html: HTML with pygments highlighting
• rst: reStructuredText
• texpweave: Latex output with user defined formatting using named environments (in latex header)
• texminted: Latex with predefined minted environment for codeblocks