# How should I have found out that LuaTeX removed \write18?

I'm trying very hard not to make this sound like a rant, but I'm very angry right now.

I was trying to use the TikZ externalization to create a scatter plot for my thesis. There were too many data points for TeX's memory so I switched to LuaTeX. The externalization didn't work.

(MWE like:

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize
\begin{document}
\tikz\node{Hello};
\end{document}


)

I had past experience with MiKTeX that suggested that changing -shell-escape to -enable-write18 can make a difference, as well as -enable-write18 vs --enable-write18. I spent literally half a day (remember, LuaTeX compilation times are still orders of magnitude slower on MiKTeX) trying to make it work and figure out why it doesn't, changing the command line around as well as the value of tikz/external/shell escape. Nothing worked.

Today I do a Google search and I find the answer at https://github.com/gpoore/minted/issues/112: There is no more \write18 in LuaTeX. You have to \usepackage{shellesc} which adds it back again.

My question is, I'm trying to be a good user here and always RTFM. Where is TFM I should have read to find out that this functionality was removed and how to add it back? To avoid spending half a day thinking I'm doing something wrong? Surely it's not the Github issue of a completely unrelated package?

• Drop by in chat and see the LaTeX team rant about the newest LuaTeX features? – Johannes_B Apr 23 '17 at 8:52
• \write18 was never part of TeX itself, it's an extension using the fact that anything above 15 is 'free' in standard TeX. LuaTeX is quite deliberately making breaking changes, and up to v1.0 was 'beta' quality only. Given that \write18 is a security risk, the LuaTeX devs felt that it's a 'edge' feature (it's not \def or similar!). Various macro package developers have been working hard to make the changes transparent to end users, but this is non-trivial. I'm not sure what an answer here would be. – Joseph Wright Apr 23 '17 at 8:55
• The LuaTeX manual, on page 44 (as of currently) says "Also, we no longer support write18 because os.execute can do the same." The manual also mentions --enable-write18 and --disable-write18 flags though. But see also the warning at LuaTeX documentation page: “The snapshot manual (in pdf format) is automatically updated from the source […] but nothing in it should be considered stable. The way things are right now, the actual binary is the definitive answer to what luatex does or does not do.” – ShreevatsaR Apr 23 '17 at 9:00
• @marczellm The LuaTeX manual has always been clear about it being beta software (up to v0.95). I'm not sure what Christian Schenk could do about the change here: MiKTeX (unlike TeX Live) does allow changes in engine binaries within the same release. (TeX Live users will only get the update by changing version.) As I've said, from the point of view of the LuaTeX developers, shell escape is not a core feature: they have after all changed the hyphenation approach compared to TeX90! – Joseph Wright Apr 23 '17 at 9:04
• See also “Can I use LuaTeX today?” on the LuaTeX FAQ page. Here's a screenshot. As I understand it, the LuaTeX team has always been clear about it being in flux, and when they feel comfortable about changing that statement they will do so. (Disclaimer: I am only an occasional user of LuaTeX and all I know about it is from clicking around the web site… but such a note has always been there AFAIK.) – ShreevatsaR Apr 23 '17 at 9:12

## 3 Answers

It's a bit unfortunate that Christian Schenk deploys in MiKTeX new versions of the binaries (luatex in this case) as soon as they are available.

TeX Live only changes (core) binaries with the yearly release and doesn't modify them during the year (except for very serious bugs, but these are usually caught during the pretest phase).

When LuaTeX 0.87 was released, many changes were made, in particular the one about \write18. If you look at LaTeX News 24, released February 2016, you'll see

and also

• Not quite sure why you blame miktex handling of binaries, the example doesn't work in TL17 pretest either. There too you need to add shellesc and unless shellesc is added to the latex kernel similar questions will pop up for texlive when 2017 gets official. Als shellesc was added to miktex month ago. – Ulrike Fischer Apr 23 '17 at 10:19
• @UlrikeFischer I don't blame MiKTeX; I just observe that the release of TL 2016 had no problems, provided shellesc is loaded as recommended in the newsletter. – egreg Apr 23 '17 at 10:24
• With shellesc it works also with miktex. There is no binary problem here. Tl16, TL17 and miktex behave here quite identical: you need shellesc and shellesc is avalaible. – Ulrike Fischer Apr 23 '17 at 10:28
• I accepted this answer because the newsletter is much easier to follow for breaking changes than a mailing list or a 230 page manual. – marczellm Apr 24 '17 at 13:32

The manual of luatex mentions the change:

Also, we no longer support write18 because os.execute can do the same.

But I doubt very much that even if you had read this that you would have understand the implications and that it could affect tikzexternalize.

The change happened last year and if you search this site for shellesc you will find a number of questions (also involving texlive, it is not a miktex specific problem, Cannot run tikz externalize with lualatex but it used to work). But again: even if you saw one of this questions during the year I wouldn't expect from you to remember them and realize that they applicate to your current problem.

So what could you do? Some tipps:

• Changes happen, even more with new software like luatex. Accept this.
• Not every change is obvious and can be learned by reading the manual.
• Don't try to solve your problems for hours alone. Search a bit and if you have the impression that you don't make any progress then better spent your time to make a sensible minimal example and then ask.

If using a beta release of software it's best to follow its development channels as breaking changes are expected in beta releases. This was discussed on the luatex list before the change was made. (Not true of all breaking changes in luatex:-). For example this thread from November 2015 http://tug.org/pipermail/luatex/2015-November/005499.html

Without following the mailing list, this change was flagged in ltnews and also in the luatex manual.

That said, in an ideal world an end user would not need to know this at all, we did search ctan for every use of \write18 that we could find and try to reach out to package developers to update packages before the changed luatex was distributed in major distributions. The minted github issue that you referenced being one example of that.

That there are still packages that require the end user to use shellesc (or luatex85) packages is a bit unfortunate but a possibly unavoidable result of the distributed nature of LaTeX package development. Packages are updated as and when the developers have time or inclination to publish an update.