Using \input|texlua... to emulate \directlua in pdfTeX

Question

When --shell-escape is enabled, pdfTeX allows to run external programs and input the result expandably with \input|"...". Together with the Lua interpreter texlua, this could be used to provide a dumbed down version of \directlua. However, texlua expects a file name as its argument.

In my setup (Ubuntu 10.04, with up-to-date TeXLive 2012 install), one can run pdftex --shell-escape on

\everyeof{\noexpand}
\message{\input|"echo print\string\(3.4+5.7\string\) > tmp.lua ; texlua tmp.lua" a}
\bye

to get \message{9.1}. Note that the \everyeof{\noexpand} is in part for convenience, it would be possible to make the construction work (not entirely robustly) without this non-expandable step.

This approach is not portable because it relies on echo, which probably doesn't exist in Windows environments. One way around this would be to have a simple Lua script taking the code print\string\(3.4+5.7\string\) as its argument (see a recent question on texlua arguments), then somehow performing it. Is this possible? Otherwise, we need an os-dependent approach.

A second problem is that special characters such as parentheses need escaping. Which ones? In particular, I'm worried about quotes ", which are used to enclose the argument of \input|. Given a list of which escapings to perform, I have some code up my sleeve to do that expandably (see the l3kernel-extras package on the LaTeX3 svn repository).

Basically, my question boils down to: "How can I invoke texlua from pdfTeX with arbitrary Lua code as portably as possible?"

Answer 1

The echo built-in is available on both Unix and Windows, and so provided platform detection is available then the problem is in principal one of deciding which characters to escape. For LaTeX, platform detection can be done using the ifplatform package, which provides the \ifwindows switch.

The first character to worry about is the 'chain process' one. On Windows, this is &, whereas on Unix ; is correct. That be handles with something like

\RequirePackage{ifplatform}
\edef\luachainprocess{\ifwindows\string&\else;\fi }

There is then the question of special characters in the echo part. For Windows, the required substitutions are

• > replaced by ^>
• < replaced by ^<
• | replaced by ^|
• & replaced by ^&
• ^ replaced by ^^

with everything else left alone by echo.

On Unix, the appropriate list seems to be

• > replaced by \>
• < replaced by \<
• ( replaced by \(
• ) replaced by \)
• & replaced by \&
• | replaced by \|
• ` replaced by \`
• \ replaced by \\

(Note: that is with the bash shell, but others may vary.)

As Windows allows for example ^( for (, it would probably be easiest to use a common set of characters for replacement with only the escape character itself varying. (The same is true for Unix with for example \^.)

One thing that is not clear at all is if there is any way to deal with " in either case, as the pipe syntax for \input does not provide an escape for this, so it seems to be misunderstood whatever you do. This is particularly frustrating for the Unix case as generating effectively

 echo "<code>"

would avoid needing to escape anything other than " inside the code.

Answer 2

I found out about the loadstring function in Lua, which converts a string to Lua code. To avoid any escaping issue for the TeX → Lua conversion, the TeX code converts the argument of \texlua to a space-separated list of character codes, which is then given as extra arguments to texlua. The Lua code decodes this using to str.char.

The remaining weak link is that \input|..., just like any other \input or \scantokens, inserts an end-of-file marker, which may not appear in the argument of any command. We need to get rid of it, and one way is to add some material just before, and just after, the end of the file, here \romannumeral\numexpr0\noexpand and \relax (the \noexpand is key). Unfortunately, the code \romannumeral\numexpr0\noexpand is not inserted if there is a Lua error; in this case, the end-of-file marker is not removed, and all hell breaks loose.

\begingroup
  \catcode `@ = 11
  %
  % Get \texlua@input = primitive \input
  \expandafter\ifx\csname @@input\endcsname\relax
    \global\let\texlua@input\input
  \else
    \global\let\texlua@input\@@input
  \fi
  %
  % Write very simple Lua file.
  \newwrite \texlua@write
  \xdef \texlua@file {\jobname.texlua}
  \immediate \openout \texlua@write \texlua@file \relax
  \immediate \write \texlua@write
    {%
      assert(loadstring(string.char(unpack(arg))))()%
      \detokenize{print("\\romannumeral\\numexpr0\\noexpand")}%
    }
  \immediate \closeout \texlua@write
  %
  % Detokenize the argument and convert spaces to category other.
  \gdef\texlua@str#1%
    {%
      \expandafter\texlua@str@i
        \detokenize{#1}\\ \\\texlua@str@ii/ \relax
    }
  \lccode `* = 32
  \lowercase{\gdef\texlua@str@i#1 #2\\#3 {#3\texlua@str@i#1*#2\\{#3} }}
  \gdef\texlua@str@ii/#1#2\\#3\relax{#2}
  %
  % Convert each character to its character code, space-separated.
  % This is not "f-expandable", it is "restricted expandable" in
  % expl3 terms.
  \gdef\texlua@num#1%
    {\expandafter\texlua@num@i\romannumeral-`q\texlua@str{#1}\relax}
  \gdef\texlua@num@i#1%
    {%
      \ifx #1\relax
      \else
        \number `#1 \space
        \expandafter\texlua@num@i
      \fi
    }
  %
  % Calling `\input|`.
  \gdef\texlua#1%
    {%
      \texlua@input|"texlua \texlua@file \space \texlua@num{#1}"%
      \relax
    }
\endgroup

\message{a\texlua{io.write(3+4)}b}

Answer 3

In filter module in ConTeXt provides the necessary boilerplate to run the content of an environment or macro through an external command (full disclosure, I am the author of the module). It does not use \input| to read the output, rather assumes that the external program will write the file to an external file and then the user can specify how to read the file (the default is \ReadFile, which is similar to \input).

The example below illustrates how to process using texlua.

% engine=pdftex

\usemodule[filter]

\defineexternalfilter
    [luatex]
    [filtercommand={texlua \externalfilterinputfile > \externalfilteroutputfile}]

\starttext

\inlineluatex{print((3+5)*3)}

\startluatex
  local t = {1,2,3}
  print("Number of entries " .. #t)
\stopluatex

\stoptext

Note: The \inlineluatex macro is not expandable.