Workaround for open lualatex bug in \scantokens?

As package author I am in need of writing a manual with lots of examples.

Now I started to make heavy use of lualatex in order to speed up operations which take a long time in TeX (of my package, not the manual) -- and I stumbled about an open ticket (bug?) of lualatex which blocks me when I attempt to activate my code changes in the manual.

Background:

An example in my manual should be something like

\begin{codeexample}[]
<code here>
\end{codeexample}


and should result in (1) the listing in some verbatim output and (2) the result of executing <code>. I want to type the listing once (and once only).

This is a good use-case for \scantokens : I collect the tokens with category code 12 (other) in order to show the verbatim output as code listing. This listing will include syntax highlighting and automatic cross references. Afterwards I use \scantokens in order to reset the category codes to their initial meaning such that I can execute the code (to show the outcome).

This last invocation of \scantokens works flawlessly in pdftex. It does not work at all in lualatex.

The issue is known, see http://tracker.luatex.org/view.php?id=733 and \scantokens in LuaTeX .

While I could write a lua version of my package and translate the manual with the pdftex version, I would rather see the effects and document special cases. Eventually, the lua version might offer unique capabilities. Consequently, I do not want to wait until someone else fixes that open ticket - and I am searching for workarounds.

Does anyone know a workaround?

The problem as such can be formulated very simple as pointed out in the linked tex.se question. I would formulate it as

\documentclass{standalone}

\begin{document}

Here should come text: \scantokens{^^JNamely this text}

\end{document}


pdftex results in

whereas luatex results in

which is the same as in the linked question.

But I need the newlines in my use case.

This here is a reduced (I suppose minimal) example of my use-case in which I collect a listing verbatim and reassign catcodes afterwards in order to execute the listing. This "execute the listing" needs the newlines, otherwise it will fail to work. And: yes, I do not typeset the listing here (it is a minimal example).

\documentclass{article}

\usepackage{pgfplotstable}

\makeatletter
% Define \find@example such that it doesn't destroy catcodes:
\begingroup
\catcode|=0
\catcode[= 1
\catcode]=2
\catcode\{=12
\catcode \}=12
\catcode\\=12 |gdef|find@example#1\end{codeexample}[|endofcodeexample[#1]]
|endgroup

\def\OLDNEWLINE{^^J}%
%% ATTEMPT (1*):
%% This here result in output of the first minimal. But it breaks the
%% second one.
%%\def\OLDNEWLINE{}%

% define \returntospace.
%
% It should define NEWLINE as {}, spaces and tabs as \space.
\begingroup
\catcode\^=7
\catcode\^^M=13
\catcode\^^I=13
\catcode\ =13%
\gdef\returntospace{\catcode\ =13\def {\space}\catcode\^^I=13\def^^I{\space}\catcode\^^M=13\def^^M{\OLDNEWLINE}}%
%
% ATTEMPT (2*):
%\gdef\returntospace{\catcode\ =13\def {\space}\catcode\^^I=13\def^^I{\space}\catcode\^^M=13}%
\endgroup

\def\codeexample[#1]{%
\parindent0pt
\begingroup%
\par%
\medskip%
\let\do\@makeother%
\dospecials%
\obeylines%
\@vobeyspaces%
\catcode\^^M=13 %
\find@example}
\def\endofcodeexample#1{%
\endgroup%
{%
\returntospace%
}%
%
% ATTEMPT (2*): This here fixes the first minimal example together with (2*):
%\catcode\^^M=9 % 9 == ignore
\expandafter\scantokens\expandafter{\code@temp}%
\end{codeexample}
}

\makeatother

\begin{document}

This should result in the picture:
\begin{codeexample}[]
\begin{tikzpicture}
\draw[red,->] (0,0) -- (1,1);

\draw[green] (0,1) -- (1,0);
\end{tikzpicture}
\end{codeexample}

This should result in the table:
\begin{codeexample}[]
\pgfplotstabletypeset{%
A B
2 3
4 5
}
\end{codeexample}
\end{document}


pdftex generates the expected outcome which is

whereas the luatex output is

I already tried a couple of things as outlined in the code. I already had the idea to use

\scantokens{\def\CONTENT{<sequence of 'other' catcodes}}


and use a "search-and-replace" routine on \CONTENT to replace every occurence of newlines with - well, I did not know with what. And simple search-and-replace routines will fail with curly braces inside of this macro.

I have used Tex live 2013 (2014 does not fit on my hard drive currently). The lua ticket is in state "new" so I do not expect useful updates.

So: does anyone have an idea how I could adopt this codeexample code in order to (a) collect the listing of both examples and (b) execute both listings successfully without modifying the listings as such?

• Can't you write the code like showexpl does it to an external file? Or change the catcodetable in luatex? – Ulrike Fischer Aug 24 '14 at 11:09
• @UlrikeFischer the idea with a tmp file would work, right (maybe that should become an answer)! I am unfamiliar with luatex's catcodetables. Can you achieve the desired effect with them? I would like to hear about it (sounds like less overhead) – Christian Feuersänger Aug 24 '14 at 18:15
• I don't have the time to look about the catcode table (just came back and have a lot to do), but imho they should be a better way if you don't need a pdflatex solution. – Ulrike Fischer Aug 24 '14 at 18:24
• Ok, thanks for the note. I'll do some research next week. Sounds like a better approach, yes. Temp files are always ugly... – Christian Feuersänger Aug 24 '14 at 18:29
• I have managed to experiment a lot: the idea to use something of sorts \directlua{local var=[==[\code@temp]==]; io.write("processing " .. var .. "-------\noexpand\n"); tex.sprint(0, var)} is quite good... however, it handles newline characters in a very confusing way. I will file a new question in order to understand that stuff. For now it seems as if writing the code into a tmp file is the best solution. Do you want to add it as answer? – Christian Feuersänger Aug 30 '14 at 14:59

Instead of \scantokens\expandafter{\code@temp}, you can pass the string to Lua, let it split the string at new lines, and tex.print each line. Calling tex.print only once would result in a single line of input, and no amount of catcode trickery would make TeX behave as if there were multiple lines.

I thus use the following Lua code in the full example below:

local s = "\luaescapestring{\code@temp}"
for line in s:gmatch("[^\string\n]+") do
tex.print(line)
end


As it is, the code is not quite a replacement for \scantokens. A more accurate replacement would be the following, which takes care of the value of \newlinechar. If it is negative (well, I should also test for when it is too big) then output the string directly. Otherwise, set the variable newline to the new line character (as a Lua string), possibly with an escaping %, then define a pattern which matches any sequence of one or more characters other than a new line (actually I just realize that this will fail when there are two new lines in a row), and print each such sequence through tex.print.

\long\def\myscantokens#1{\directlua{%
local s = "\luaescapestring{\detokenize{#1}}"
if \the\newlinechar < 0 then
tex.print(s)
else
local newline = string.char(\the\newlinechar)
if not newline:match("\string\%w") then
newline = "\string\%" .. newline
end
local pattern = "[^" .. newline .. "]+"
for line in s:gmatch(pattern) do
tex.print(line)
end
end}}


Anyhow, here I'll use the simpler code since it seems to work for your case.

\documentclass{article}

\usepackage{pgfplotstable}

\makeatletter
% Define \find@example such that it doesn't destroy catcodes:
\begingroup
\catcode|=0
\catcode[= 1
\catcode]=2
\catcode\{=12
\catcode \}=12
\catcode\\=12 |gdef|find@example#1\end{codeexample}[|endofcodeexample[#1]]
|endgroup

\def\OLDNEWLINE{^^J}%

% define \returntospace.
%
% It should define NEWLINE as {}, spaces and tabs as \space.
\begingroup
\catcode\^=7
\catcode\^^M=13
\catcode\^^I=13
\catcode\ =13%
\gdef\returntospace{\catcode\ =13\def {\space}\catcode\^^I=13\def^^I{\space}\catcode\^^M=13\def^^M{\OLDNEWLINE}}%
\endgroup

\def\codeexample[#1]{%
\parindent0pt
\begingroup%
\par%
\medskip%
\let\do\@makeother%
\dospecials%
\obeylines%
\@vobeyspaces%
\catcode\^^M=13 %
\find@example}
\def\endofcodeexample#1{%
\endgroup%
{%
\returntospace%
}%
\directlua
{%
local s = "\luaescapestring{\code@temp}"
for line in s:gmatch("[^\string\n]+") do
tex.print(line)
end
}%
\end{codeexample}
}
\makeatother

\begin{document}

This should result in the picture:
\begin{codeexample}[]
\begin{tikzpicture}
\draw[red,->] (0,0) -- (1,1);

\draw[green] (0,1) -- (1,0);
\end{tikzpicture}
\end{codeexample}

This should result in the table:
\begin{codeexample}[]
\pgfplotstabletypeset{%
A B
2 3
4 5
}
\end{codeexample}
\end{document}

• Two comments: In a LaTeX document, one should probably use "\luatexluaescapestring{\detokenize{#1}}" rather than "\luaescapestring{\detokenize{#1}}". Second, to avoid having to use this dreadful macro name, you could load the luacode package and simply type \luatring{\detokenize{#1}}. Note: a side benefit of using \luastring is that the double quotes that must surround \luatexluaescapestring can also be dispensed with. – Mico Sep 2 '14 at 19:38
• Tricky. Seems as if my approaches in tex.stackexchange.com/questions/198674/… have been the right direction... I should have applied it to the codeexample stuff rather than playing with the minimal example in the linked question. Thanks a lot. – Christian Feuersänger Sep 2 '14 at 20:28
• For the record, I've replied to Christian's question there. – Bruno Le Floch Sep 3 '14 at 12:02
• For the records: I have used a slightly modified approach which also keeps empty lines. – Christian Feuersänger Sep 28 '14 at 18:23