Workaround to make catchbetweenfiletags work with lualatex

Question

I recently discovered that a basic example of the package catchbetweenfiletags doesn't work with lualatex (while it works with pdflatex).

What follows is the MWE. Do you know if there is workaround to make it work with lualatex?

File catch1.tex:

\documentclass{article}
\usepackage{catchfilebetweentags}
\begin{document}
test
\ExecuteMetaData[catch2]{mytag}
\end{document}

File catch2.tex:

\documentclass{article}
\begin{document}
%<*mytag>
CATCH2
%</mytag>
\end{document}

$ pdflatex catch1.tex works as expected.

$ lualatex catch1.tex gives this error:

Runaway text?
! File ended while scanning text of \CatchFBT@tok.
<inserted text> 
}
l.7 \ExecuteMetaData[catch2]{mytag}

? 

Versions:

pdfTeX 3.14159265-2.6-1.40.15 (TeX Live 2014)

This is LuaTeX, Version beta-0.79.1 (TeX Live 2014) (rev 4971)

catchfilebetweentags : 2011/02/19 v1.1

For reference, using the package's command \CatchFileBetweenTags fails in the same way. From the comment by @egreg, it also works with xelatex.

Answer 1

Imho it is the same problem than this one: \scantokens in LuaTeX and this http://tug.org/pipermail/lualatex-dev/2011-November/001376.html. The package changes \endlinechar and so luatex slides over the end of the file.

One can avoid the error message by disabling the \endlinechar change:

\documentclass{article}
\usepackage{catchfilebetweentags,ifluatex}
\makeatletter
\def\CatchFBT@sanitize{%
   \@sanitize
   \@makeother\{%
   \@makeother\}%
   \ifluatex\else
   \endlinechar=`\^^J%
   \fi
}% 
\makeatother
\begin{document}
test
\ExecuteMetaData[catch2]{mytag}
\end{document}

One can also set ^^J to be an "end of line char".

\documentclass{article}
\usepackage{catchfilebetweentags,ifluatex}
\makeatletter
\def\CatchFBT@sanitize{%
   \@sanitize
   \@makeother\{%
   \@makeother\}%
   \endlinechar=`\^^J%
   \ifluatex
   \catcode`\^^J=5
   \fi
}% 
\makeatother
\begin{document}
test
\ExecuteMetaData[catch2]{mytag}

\end{document}

But both changes the output compared to pdflatex if the catched file contains empty lines.

Answer 2

I tried various things to go around the buggy \scantokens from LuaTeX. In the end the most robust appears to me (for the time being) to just drop using \scantokens in that context and go through a real intermediate file. Here is the code:

% \scantokens is buggy in LuaTeX
% http://tex.stackexchange.com/questions/65151/scantokens-in-luatex
% http://tug.org/pipermail/lualatex-dev/2011-November/001376.html

\documentclass{article}
\usepackage{filecontents}
\begin{filecontents*}{catch2}
\documentclass{article}
\begin{document}
%<*mytag>
CATCH2
SECOND LINE

THIRD LINE AFTER ONE EMPTY LINE


FOURTH LINE AFTER TWO EMPTY LINES
%</mytag>
\end{document}
\end{filecontents*}
\usepackage{catchfilebetweentags}
\usepackage{ifluatex}

% CIRCUMVENTING LUALATEX BUGGY \scantokens

\makeatletter

\ifluatex 
   \newwrite\CATCHPATCHOUT
\fi

\def\CatchFBT@Fin@l#1[#2]{%
   \begingroup
      \endlinechar\m@ne \makeatletter #2%
      \ifluatex
         \immediate\openout\CATCHPATCHOUT=\jobname.catchpatch
         \immediate\write\CATCHPATCHOUT {\CatchFBT@tok{\the\CatchFBT@tok}}%
         \immediate\closeout\CATCHPATCHOUT
         \@@input \jobname.catchpatch
      \else
      \scantokens\expandafter{%
         \expandafter\CatchFBT@tok\expandafter{\the\CatchFBT@tok}}%
      \fi
      \CatchFBT@IsAToken{#1}
         {\global#1\expandafter{\the\CatchFBT@tok}}
         {\xdef#1{\the\CatchFBT@tok}}%
      \ifx\CatchFBT@tok#1\else\global\CatchFBT@tok{}\fi
   \endgroup
}% \CatchFBT@Final
\makeatother

\begin{document}\thispagestyle{empty}
\ttfamily


\CatchFileBetweenTags\tmp{catch2}{mytag}

\meaning\tmp

\noindent\ExecuteMetaData [catch2]{mytag}

\end{document}

On this example, the following is obtained, be it with pdfTeX or LuaLaTeX:

Notice that this is the behavior of package catchfilebetweentags: if you want to reinstore line breaks you should use the final optional argument to \CatchFileBetweenTags to modify \endlinechar from its -1 setting.

Example:

\CatchFileBetweenTags\tmp{catch2}{mytag}[\endlinechar=13 ]

\meaning\tmp

Again we obtain the same result with both engines:

Notice that the space before CATCH comes from the end of line of the tag line. This is what the package does (nothing to do with my patch here).