Issues with macro expansion/lstlisting after updating to LuaLaTeX Version 2021-12-31

Question

I am using Archlinux with texlive. After updating my installation on 2022-01-02 to the newest Archlinux texlive-packets, some features of LuaLaTeX seem to have changed significantly. For example, tex.sprint(STRING) does no longer work inside of the file name argument of \lstinputlisting{FILENAME}.

Before the update, the following MWE was working:

\documentclass{minimal}

\usepackage{luacode}
\usepackage[procnames]{listings}

\begin{filecontents*}{before-file1_test_label.lst}
    \LaTeX~macros work
\end{filecontents*}

\newcommand{\lstSHAName}[2]{before-\directlua{
            local string = "#1"
            tex.print(string .. '_test')
        }_#2.lst}
    
\begin{document}
    %\lstinputlisting{\lstSHAName{file1}{label}} % was working before, now crashes
    \lstinputlisting{before-file1_test_label.lst} % still works, but requires setting path manually 
\end{document}

If I compile with the new texlive version, the compilation halts and the following error message is shown:

! Missing \endcsname inserted.
<to be read again> 
global 
l.16    \lstinputlisting{\lstSHAName{file1}{label}}

What additionally raises the suspicion that the issues are related to the update is the fact that my co-worker with MikTeX on Windows still can compile the old code without any issues.

Work-around attempt

The idea was to use the tex.sprint(STRING)-functionality of LuaLaTeX to print the \lstinputlisting command with all options and afterwards let it get executed by LaTeX.

\documentclass{minimal}

\usepackage{luacode}
\usepackage[procnames]{listings}

\begin{filecontents*}{before-file1_test_label.lst}
    \LaTeX~macros work
\end{filecontents*}

\newcommand{\printListing}[3]{%
        \directlua{
            local strA = "#1"
            local strB = "#2"
            local strC = "#3"
            local string = "\\lstinputlisting[caption={" .. strC .. "}]{before-" .. strA  .. "_test_" .. strB .. ".lst}" 
            tex.sprint(string)
    }
}
    
\begin{document}
    \printListing{file1}{label}{Test caption} % works
    %\printListing{file1}{label}{Test caption with \LaTeX~macro} % throws an error due to expansion
    \printListing{file1}{label}{Test caption with \\LaTeX{} macro} % works, but tilde would have also to be escaped differently
\end{document}

Passing LaTeX macros into \directlua without escaping (line 22) leads to the error

! Undefined control sequence.
\S@10 ->\gdef \tf@size 
                       {10}\gdef \sf@size {7}\gdef \ssf@size {5}
l.22

Escaping manually is somewhat possible. However, the original idea was to provide a wrapper macro for inexperienced LaTeX users. Having to know how to escape in LaTeX so that lua will process it correctly defeats the original purpose.

Questions

1. Can this issue be solved by type conversion/expansion/escaping before piping the file path to \lstinputlisting? And if so, how (could not get it to work; always got the same error)?
2. Is there for my work-around attempt a way with expl3 to pre-process the arguments before passing them on to \directlua, so that lua does not expand the macros (for example passing on \\LaTeX instead of \LaTeX) ?

Installed texlive packages

Packages I think are relevant for the problem:

• texlive-bin 2021.5945-1, build date: 2021-12-27
• texlive-core 2021.61403-1, build date: 2021-12-27
• texlive-formats-extra 2021.57972-1, build date: 2021-04-06
• texlive-latexextra 2021.61405-1, build date: 2021-12-27

Answer 1

1.

In the general case, no. By the rule of TeX parsing, if you have

\lstinputlisting{\lstSHAName{file1}{label}}

then TeX will expands \lstinputlisting first passing the argument {\lstSHAName{file1}{label}} (after removing braces) into it, and there's no guarantee that the \lstinputlisting macro will ever expand the argument.

However it remains possible to redefine \lstinputlisting (can be fragile and interfere with the listings package!).

Alternatively you can also do something like

\injectlstSHAName{
\lstinputlisting{\lstSHAName{file1}{label}}
}

where the outer macro reads the argument (verbatim if you want to allow users to use catcode-changing commands inside), do the appropriate replacement, then print back the content.

2.

One way is to read the argument verbatim (see part 1 of code below).

Another way is detokenize it. In this case it's the same anyway because the caption is grabbed as an argument (i.e. \lstinputlisting[caption={a {\verb+123+} b}]{before-file1_test_label.lst} will not work either way. It remains possible to make it work with cprotect, however.)

Yet another way is to not round-trip it through Lua at all and just leave it in TeX. This method has the advantage of preserving the tokens catcode, just in case the user is knowledgeable and pass weird catcode combinations into the command.

%! TEX program = lualatex
\documentclass{minimal}

\usepackage{luacode}
\usepackage[procnames]{listings}

\begin{filecontents*}{before-file1_test_label.lst}
    \LaTeX~macros work
\end{filecontents*}

    
\begin{document}

    % ======== method 1
\NewDocumentCommand{\printListing}{mmv}{%
        \directlua{
            local strA = "#1"
            local strB = "#2"
            local strC = "\luaescapestring{#3}"
            local string = "\\lstinputlisting[caption={" .. strC .. "}]{before-" .. strA  .. "_test_" .. strB .. ".lst}" 
            tex.sprint(string)
    }%
}

    \printListing{file1}{label}{Test caption}
    \printListing{file1}{label}{Test caption with \LaTeX~macro}

    % ======== method 2
\RenewDocumentCommand{\printListing}{mmm}{%
        \directlua{
            local strA = "#1"
            local strB = "#2"
            local strC = "\luaescapestring{\detokenize{#3}}"
            local string = "\\lstinputlisting[caption={" .. strC .. "}]{before-" .. strA  .. "_test_" .. strB .. ".lst}" 
            tex.sprint(string)
    }%
}

    \printListing{file1}{label}{Test caption with \LaTeX~macro}

    % ======== method 3
\RenewDocumentCommand{\printListing}{mmm}{%
    \lstinputlisting[caption={#3}]{before-#1_test_#2.lst}
}

    \printListing{file1}{label}{Test caption with \LaTeX~macro}

\end{document}

Turns out method 3 is the simplest.