TikZ externalize while using -output-directory

Question

I'm using tikzexternalize to speed things up and to have a clean working directory, And I'm using -output-directory on compilation to have a clean working directory (actually I use cluttex, but I've seen the problem with just -output-directory as well).

So my tex code looks something like this

\documentclass{article}

\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize[prefix=figures/] % activate and define figures/ as cache folder

\begin{document}

\begin{tikzpicture}
    \node[draw] {lore ipsum};
\end{tikzpicture}

\end{document}

and compile with pdflatex -shell-escape -output-directory='tex-aux' main.tex (actually I'm using LuaLaTeX, but the problem remains when using normal (pdf)latex).

The error I get is

 \write18 enabled.
entering extended mode
! I can't write on file `figures/main-figure0.log'.
(Press Enter to retry, or Control-D to exit; default file extension is `.log')
Please type another transcript file namesystem returned with code 256

! Package tikz Error: Sorry, the system call 'pdflatex -shell-escape -halt-on-e
rror -interaction=batchmode -jobname "figures/main-figure0" "\def\tikzexternalr
ealjob{main}\input{main}"' did NOT result in a usable output file 'figures/main
-figure0' (expected one of .pdf:.jpg:.jpeg:.png:). Please verify that you have
enabled system calls. For pdflatex, this is 'pdflatex -shell-escape'. Sometimes
 it is also named 'write 18' or something like that. Or maybe the command simpl
y failed? Error messages can be found in 'figures/main-figure0.log'. If you con
tinue now, I'll try to typeset the picture.

See the tikz package documentation for explanation.
Type  H <return>  for immediate help.
 ...

l.12 \end{tikzpicture}

So I know when removing -output-directory='tex-aux' everything works just fine (I even tried manually creating the folder tex-aux/figures just for the case this is the trouble, but it didn't help).

I also know that the hint to check if system calls are enabled shouldn't be the problem, since they are enabled with -shell-escape.

Any suggestions on what the problem might be (and how to solve it)?

Answer 1

Solution1: using cluttex --output-directory=tex-aux -e lualatex -shell-escape main.tex with

\documentclass{article}

\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize[prefix=tex-aux/]

\begin{document}

\begin{tikzpicture}
    \node[draw] {lore ipsum};
\end{tikzpicture}

\end{document}

but it's important that prefix equals output-directory and the folders output-directory and output-directory/prefix exist. So far this looks to me like the cleanest way (not much configuration effort) but I'm not sure if this will work on all setups because we`re fooling the extern lib with the prefix.

Solution2: use something like

cd tex-aux && \
cp ../main.tex ./ && \
lualatex -recorder -halt-on-error -interaction=nonstopmode -file-line-error -shell-escape -jobname='main' "\makeatletter\def\input@path{{..}}\makeatother\input{main.tex}" \
&& cp $@ ../

($@ for the output file comes from make)

Here we're adding the parent directory to the path (see https://tex.stackexchange.com/a/24827/206293) so that `main.tex´ is found not only on the direct call in the compilation command but also in the compilation commands issued by tikz-extern lib.

Solution3: Looking at the cluttex code normally it should modify TEXINPUTS (add the directory before the cd) in combination with the cd approach when using the flag --change_directory. Currently this doesn't work completely, but one might use this approach manually (or maybe this pull request get through and one can use cluttex directly)