Is it possible to call external programs from LaTeX?

I'm trying to write very specific LaTeX package that does a lot of complex computation. It encloses some parts of text (or whatever else user wants to print) into boxes and then runs quite complex computations to determine where to lay out those boxes in the document.

It is not practical to program that computation in LaTeX - they will be challenging even in high-level programming language. Is there a way to delegate that computations to some external program? The desired workflow would be like this:

1. In LaTeX code collect necessary parameters: dimensions of all the boxes to be print, textwidth and textheight.
2. Send it to external program.
3. Based on received parameters the program would compute placement of all the boxes and send the results back to LaTeX somehow.
4. LaTeX would use externally computed positions of boxes and print stored boxes at those locations.

Is there a way to communicate with external programs? It does not have to be anything elegant, storing parameters in file and then reading computed results from another file would be sufficient. I do not plan to redistribute the package yet so it does not matter that the external program cannot be redistributed.

-
It is possible with write18 feature. But what you actually plan to do, I don't know. If you search this site, you will get codes where write18 is used. I use it to run convert (of imagemagick) from within the latex document. – Harish Kumar Jan 14 '13 at 3:28
Look into the Python package and Sage package. They rely on write18 and I think they can do what you want. – DJP Jan 14 '13 at 3:38
Depending on how complicated the calculations are, using LuaTeX might even be an option. – Scott H. Jan 14 '13 at 4:51
This sound like a job for LuaTeX. See here for an example what you can do with it – Marco Jan 14 '13 at 8:30
drasto: this answer is quite relevant I believe. – Scott H. Jan 14 '13 at 16:17

(This is a ConTeXt solution, not a LaTeX solution; I am posting this in case someone wants to achieve the same in ConTeXt).

Is there a way to communicate with external programs? It does not have to be anything elegant, storing parameters in file and then reading computed results from another file would be sufficient.

For ConTeXt, I have written a module---filter (at github)---that does exactly that. As an example, suppose you want to create figures using Graphviz and include them in ConTeXt. For that, define a graphviz environment as follows:

\usemodule[filter]

\defineexternalfilter
[graphviz]
[
filter=\externalfilterparameter{layout} -Tpdf -o\externalfilteroutputfile,
layout=dot, % neato or twopi
cache=yes,
output=\externalfilterbasefile.pdf,
]

{\externalfigure[#1]}


and then typeset a graph as:

\startgraphviz
digraph G {
a_1-> a_2 -> a_3 -> a_1;
}
\stopgraphviz


If you want to change the layout program (different algorithm used by graphviz), use:

\startgraphviz[layout=neato]
....
\stopgraphviz


You can pass arbitrary options to the external program using its command line options.

This package is just a wrapper around \write18, but provides some useful features:

• Caching of results (rerun the external program only if the content has changed)
• Specify output directory used to store all the temporary files (current directory used by default)
• Enable diagnostic information if something goes wrong.
-
Pardon my ignorance but I do not know anything about ConTeXt. Would you please provide some link to documentation and comparison with LaTeX, so I can consider this option too? – drasto Jan 14 '13 at 13:34
– Aditya Jan 14 '13 at 13:41

My pythontex package might be an option (latest version on GitHub; will be on CTAN soon). It offers the following features.

• Enter Python within your LaTeX document, and bring the output back into LaTeX.
• User-defined sessions. If you have independent calculations that are slow, just put them in separate sessions, and they will automatically run in parallel, using all available cores.
• All calculations are saved or cached; code is only executed when modified (unless you specify otherwise).
• Any errors or warnings are synchronized with your document, so that you know which line in your document caused problems.

The easiest thing to do would be to pass whatever parameters you need to Python, do your calculations, and then print LaTeX code back to the TeX side. There are a few examples in the documentation, and I've also used pythontex to answer a few questions on this site.

If you just want to get values back from Python, stored in macros, here's an example. This allows you to pass a length to Python, perform some calculations on it, and then return the output to TeX, stored in a macro. Again, usually it's probably easier to just assemble a complete block of LaTeX code within Python, and print that back instead (maybe have a template, and perform substitutions.)

\documentclass{article}

\usepackage{pythontex}

\makeatletter

% Define a Python function that takes a TeX length and a name,
% extracts the numerical part of the length and performs
% a calculation with it, and finally prints a macro definition
% back to TeX that stores the calculated value, with its
% units, using the specified name for the macro.
\begin{pylabcode}
def my_calc(length, name):
num = float(length[:-2])
unit = length[-2:]
print(r'\def\{0}{{{1}{2}}}'.format(name, str(sqrt(num**2/2)), unit))
\end{pylabcode}

\newcommand{\mycalc}[2]{%
% Create a default macro for the output.
% This avoids errors before PythonTeX runs
% and creates the real value
\expandafter\def\csname #2\endcsname{0pt}%
% When using PythonTeX commands inside macros,
% it's sometimes easiest to employ a helper
% macro to get everything expanded properly.
\expandafter\mycalc@i\expandafter{\the#1}{#2}%
}
\newcommand{\mycalc@i}[2]{%
\pylabc{my_calc('#1','#2')}%
}

\makeatother

\begin{document}

% This passes the value of \textwidth to Python,
% performs a calculation on it, and stores the result,
% including units, in the macro \returnvalue
\mycalc{\textwidth}{returnvalue}

\returnvalue

\end{document}

-
+1 I like this as well. There are now many options for me to choose from, I'm slowly exploring them one after another and trying to choose the one that suits me best. – drasto Jan 15 '13 at 7:48
After reading again: passing required values to python as parameters of macro is not really an option in my case. Python code needs to access widths and heights of number of saved boxes, and I do not know in advance how many of them there will be. So is there a way to access width and height of a particular saved box within python code? (equivalent in LaTeX would be \wd\theNumberOfTheBox) – drasto Jan 15 '13 at 7:56
@drasto The only way to access TeX info in Python is to pass the TeX info to Python, which requires macros. You could create a custom macro that first saves a box and then passes the name of the box and its dimensions to Python. You could then store this information in a dictionary on the Python side, so that the name of the box is associated with its dimensions. Using that approach, you could keep track of all boxes and their dimensions on the Python side automatically, regardless of how many there are. – G. Poore Jan 15 '13 at 14:59

If you have complicated calculations, I would tend to think you want either the python package or the sagetex package. There's an example of using Python here but sagetex uses Python commands plus math commands created for the CAS known as Sage so it's ideal for complicated problems. There's a Sage Cell Server where you can experiment with Sage yourself. From here you'll find "A concise description of sagetex is that it is a collection of TeX macros that allow a LaTeX document to include instructions to have Sage compute various objects and/or format objects using the latex() support built in to Sage. So as an intermediate step of compiling a LaTeX document, all of the computational and LaTeX-formatting features of Sage can be handled automatically.". The documentation of the sagetex package states on page 5 and 6: "Note that since LaTeX will do macro expansion on whatever you give to \sage,you can mix LaTeX variables and Sage variables! If you have defined the Sage variable foo to be 12 (using, say, the sageblock environment), then you can do something like this: The prime factorization of the current page number plus foo is $\sage{factor(foo + \thepage)}$." Here's a small working example I put together showing that off:

\documentclass{article}
\usepackage{sagetex}
\pagestyle{empty}
\begin{document}
The documentation of the sagetex package gives a simple example
showing how you can combine \LaTeX and Sage variables similar
to this:
\begin{sagesilent}
foo=99
\end{sagesilent}
The prime factorization of the current page number plus foo
is $\sage{factor(foo + \thepage)}$.

The 2nd footnote (bottom of page 6) indicates there can be some
weirdness'' depending on what you're processing.
\end{document}


After running MyFile.tex with pdfLaTeX, a MyFile.sage file is created. Process this with Sage from the command line: load 'MyFile.sage' and Sage will create MyFile.sout and instruct you to LaTeX on your document again. When you do so, your document will print your document with the Sage calculations/text/pictures it created. This is the output from the code above.

It's also possible to "...get input and send it to Sage at the time of compilation" (quote from this on page 271). There's a PDF you can download with more examples of sagetex; the important application on page 9: "Make Sage write your LaTeX for you". There might be other packages that can do what you want but when if work with complicated math, sagetex is a package you need to learn.

-
Nice example thank you. I'm not familiar with Sage but I have a lot of experience with Matlab, so this might be an option as well. Is there a way to access LaTeX lengths and counters from within sagesilent environment? Especially I would need to iterate number of boxes and use their width and height in my computations. – drasto Jan 15 '13 at 7:43
As a side question is there similar package for Matlab, Haskel or even better for Prolog? – drasto Jan 15 '13 at 7:44
@drasto: I haven't used the package that way, the best place to ask is here. I'm not sure whether the pl package does what you want. – DJP Jan 15 '13 at 23:31

You could use pipes for this. Assuming you use Miktex you will have to add -enable-pipes flag to your tex command. You can then use pipes when opening files. You can open files to read and write. To read from a file, that really is a command you can use:

\newread\source
\openin \source = "|echo Text"
\loop
\ifeof \source
\global\morefalse
\else


What you do with the input (probably assign it to a length \setlength{\linewidth}{\currentline} or something)

\fi
\ifmore\repeat
\closein\source


Here echo Text would be your executed command. To write to a file or the command line input of a program works the same way, just with \newwrite, \openout, \write and \closeout. If the first character of the file is not | than Tex will look for an actual file. Be careful with escape characters though, \input{File_You_just_wrote_with_external_tool} might be easier.

-

There are a number of packages that allow this. The most general is probably bashful.

bashful provides a convenient interface to TEX’s primitive \write18—the execution of shell commands from within your input files, also known as shell escape. Text between \bash and \END is executed by bash, a popular Unix command line interpreter. Various flags control whether the executed commands and their output show up in the printed document, and whether they are saved to files.

in that you can then do anything that you can do from a BASH CLI.

If you are willing to limit yourself to Python there is the python package

This package enables you to embed Python code (www.python.org) in LaTeX and insert the output into your LaTeX document.

-
Bash is Linux only, right? Currently I'm working on Windows, is it possible to execute command line commands using bashful package, too? – drasto Jan 14 '13 at 12:17
@drasto The bashful documentation says it won't work on Windows without modification. I don't know enough about the Windows CLI to know how much modification would be required. – StrongBad Jan 14 '13 at 16:19