\sys_get_shell:nnN seems to strip newlines from shell output

Question

Goal: run a shell command and \input its output as LaTeX code.

See related: How to protect \input shell command args with backslashes

When using expl3's \sys_get_shell:nnN, newlines from the shell output seem to be replaced with spaces.
This is a problem when using certain commands like \begin{lstlisting}, which discard any text on that line up to a newline character, then process all text on future lines up to \end{lstlisting}.

For example, the following produces warnings and ignores the BOO and BAH:

\documentclass[11pt]{article}
\usepackage{listings}

\begin{document}
\begin{lstlisting}[breaklines]BAH
Failed to run:
\end{lstlisting}
\begin{lstlisting}[breaklines]BOO
-bash: notACommand: command not found
\end{lstlisting}

\end{document}

Warnings:

Package Listings Warning: Text dropped after begin of listing on input line 5.
Package Listings Warning: Text dropped after begin of listing on input line 8.

Output:

Failed to run:
-bash: notACommand: command not found

The following, using \input, works as expected, including newlines from the shell output:

\documentclass[11pt]{article}
\usepackage{listings}

\begin{document}

\input{"|OUTPUT=`notACommand \\2\\\{\\3\\( 2>&1` && echo $OUTPUT || ( ( echo \\\\\\\\begin\\{lstlisting\\}\\[breaklines\\] && echo Failed to run: && echo \\\\\\\\end\\{lstlisting\\} ) && ( echo \\\\\\\\begin\\{lstlisting\\}\\[breaklines\\] && echo $OUTPUT && echo \\\\\\\\end\\{lstlisting\\} ) )"}

\end{document}

producing desired output:

Failed to run:
sh: notACommand: command not found

However, the seemingly-equivalent use of \sys_get_shell:nnN turns newlines from the shell output into spaces:

\documentclass[11pt]{article}
\usepackage{listings}

\begin{document}

\ExplSyntaxOn
\tl_new:N \someVarForShellOutput % declare this variable
\cs_new_protected:Nn \inputFromShellCall:n {
  \sys_get_shell:nnN { #1 } { \ExplSyntaxOff } \someVarForShellOutput % disable expl syntax when parsing shell output
  \tl_analysis_show:N \someVarForShellOutput % for debugging - we see that the newlines from shell output are now spaces
  \tl_use:N \someVarForShellOutput
}
\cs_generate_variant:Nn \inputFromShellCall:n { e } % Create \inputFromShellCall:e which expands its argument before invoking the shell
\ExplSyntaxOff

\ExplSyntaxOn
\inputFromShellCall:e { OUTPUT=`notACommand~ \c_backslash_str 2 \c_backslash_str \{ \c_backslash_str 3 \c_backslash_str (~2>&1`~&&~echo~$OUTPUT~||~(~(~echo~\c_backslash_str \c_backslash_str \c_backslash_str \c_backslash_str begin\{lstlisting\} \c_backslash_str [breaklines\c_backslash_str ]~&&~echo~Failed~to~run:~&&~echo~\c_backslash_str \c_backslash_str \c_backslash_str \c_backslash_str end\{lstlisting\}~)~&&~(~echo~\c_backslash_str \c_backslash_str \c_backslash_str \c_backslash_str begin\{lstlisting\}\c_backslash_str [breaklines\c_backslash_str ]~&&~echo~$OUTPUT~&&~echo~\c_backslash_str \c_backslash_str \c_backslash_str \c_backslash_str end\{lstlisting\}~)~) }
\ExplSyntaxOff

\end{document}

The debugging \tl_analysis_show:N shows that the shell output is:

(|OUTPUT=`notACommand \2\\{\3\( 2>&1` && echo $OUTPUT || ( ( echo \\\\begin\{lstlisting\}\[breaklines\] && echo Failed to run: && echo \\\\end\{lstlisting\} ) && ( echo \\\\begin\{lstlisting\}\[breaklines\] && echo $OUTPUT && echo \\\\end\{lstlisting\} ) ))
The token list \someVarForShellOutput contains the tokens:
>  \begin (control sequence=macro:->\protect \begin  )
>  { (begin-group character {)
>  l (the letter l)
>  s (the letter s)
>  t (the letter t)
>  l (the letter l)
>  i (the letter i)
>  s (the letter s)
>  t (the letter t)
>  i (the letter i)
>  n (the letter n)
>  g (the letter g)
>  } (end-group character })
>  [ (the character [)
>  b (the letter b)
>  r (the letter r)
>  e (the letter e)
>  a (the letter a)
>  k (the letter k)
>  l (the letter l)
>  i (the letter i)
>  n (the letter n)
>  e (the letter e)
>  s (the letter s)
>  ] (the character ])
>    (blank space  )
>  F (the letter F)
>  a (the letter a)
>  i (the letter i)
>  l (the letter l)
>  e (the letter e)
>  d (the letter d)
...

Even though manually running the shell command

OUTPUT=`notACommand \2\\{\3\( 2>&1` && echo $OUTPUT || ( ( echo \\\\begin\{lstlisting\}\[breaklines\] && echo Failed to run: && echo \\\\end\{lstlisting\} ) && ( echo \\\\begin\{lstlisting\}\[breaklines\] && echo $OUTPUT && echo \\\\end\{lstlisting\} ) )

produces output with a newline before "Failed":

\begin{lstlisting}[breaklines]
Failed to run:
\end{lstlisting}
\begin{lstlisting}[breaklines]
sh: notACommand: command not found
\end{lstlisting}

https://mirror.mwt.me/ctan/macros/latex/contrib/l3kernel/interface3.pdf for \sys_get_shell:nnN states:

The ⟨shell command⟩ is converted to a string using \tl_to_str:n. Category codes may need to be set appropriately via the ⟨setup⟩ argument, which is run just before running the ⟨shell command⟩ (in a group).

\tl_to_str:n sounds like it applies to the input to the shell, but not the output, thus shouldn't be relevant here.
The category code setup here seems like it should be OK - I turn off expl3 syntax since lstlisting doesn't seem to work inside \ExplSyntaxOn (I haven't investigated why).

Edit: this can also be reproduced where the shell invocation site is not inside expl3 syntax, so no category code setup should be required:

\documentclass[11pt]{article}
\usepackage{listings}

\begin{document}

\ExplSyntaxOn
\tl_new:N \someVarForShellOutput % declare this variable
\cs_new_protected:Nn \inputFromShellCall:n {
  \sys_get_shell:nnN { #1 } { } \someVarForShellOutput % disable expl syntax when parsing shell output
  \tl_analysis_show:N \someVarForShellOutput % for debugging - we see that the newlines from shell output are now spaces
  \tl_use:N \someVarForShellOutput
}
\cs_generate_variant:Nn \inputFromShellCall:n { e } % Create \inputFromShellCall:e which expands its argument before invoking the shell

\NewDocumentCommand{\simpleNameInputFromShellCall}{m} {
  \inputFromShellCall:e { #1 }
}
\ExplSyntaxOff

\simpleNameInputFromShellCall{ OUTPUT=`notACommand \\2\\\{\\3\\( 2>&1` && echo $OUTPUT || ( ( echo \\\\begin\{lstlisting\}\\[breaklines\\] && echo Failed to run: && echo \\\\end\{lstlisting\} ) && ( echo \\\\begin\{lstlisting\}\\[breaklines\\] && echo $OUTPUT && echo \\\\end\{lstlisting\} ) ) }

\end{document}

Why are the newlines in shell output getting turned into spaces by \sys_get_shell:nnN, and can this be prevented?

Second edit:

How can I save shell output to a variable in LaTeX? talks about how \read reads balanced lines and converts newlines to spaces.
It says "To avoid this, you set \endlinechar to -1.", but that turns newlines into <nothing>.

Setting \endlinechar in the \sys_get_shell:nnN setup arg to something visible like \endlinechar=97 (which is a) successfully produces as instead of newlines. But \endlinechar=10 (the actual ascii code of the newline char) actually inserts newlines where they're present in the shell output, but seems to have the wrong category code for listlisting to recognize it.

I also tried manually modifying the catcode after setting the newlinechar:
\sys_get_shell:nnN { #1 } { \endlinechar=10 \catcode\endlinechar=5 } \someVarForShellOutput
to everything from 1 to 16, without success.
Setting it to 5 results in the newline getting replaced by a space - the same behavior without any shell setup code, which makes sense, since its default catcode (outside EXPL) seems to be 5: \showthe\catcode\endlinechar.

This language is painful lol, and I see no workaround, other than skipping \sys_get_shell:nnN and using \input{"|..."} instead.

Answer 1

Fundamentally you still need to understand how catcode in TeX works to understand what the expl3 functions are doing (for which TeXbook/TeX by topic helps).

In any case, the verbatim-like environments such as listings changes catcode while reading the content, while expl3 sys_get_shell:n tokenizes the whole content right after they're read, which makes it impossible to guarantee the correct behavior of listings.

(*) if you know exactly what catcodes are expected by listings and set those precise catcodes, it might be possible; nevertheless this implementation depends a lot on the internal implementation of listings environment.

In any case, one workaround is to get the raw content with sys_get_shell and tokenize it with scantokens (for which there's no expl3 wrapper at the moment (\tl_rescan and variants tokenize the whole thing at the start, which is not applicable here because of the same reason mentioned above), apart from \tex_scantokens:D which is not a "wrapper" at all)

\documentclass{article}
\usepackage{listings}


\begin{filecontents*}[overwrite]{test.txt}
\begin{lstlisting}
line 1  !?%#\xyz
line 2
\end{lstlisting}
\end{filecontents*}

\begin{document}

\ExplSyntaxOn
\tl_new:N \someVarForShellOutput % declare this variable
\cs_new_protected:Nn \inputFromShellCall:n {
    \sys_get_shell:nnN { #1 } { \cctab_select:N \c_other_cctab \endlinechar=10~ } \someVarForShellOutput
    \scantokens \expandafter { \someVarForShellOutput }
}
\cs_generate_variant:Nn \inputFromShellCall:n { e } % Create \inputFromShellCall:e which expands its argument before invoking the shell

\NewDocumentCommand{\simpleNameInputFromShellCall}{m} {
  \inputFromShellCall:e { #1 }
}
\ExplSyntaxOff

\simpleNameInputFromShellCall{ cat test.txt }

\end{document}

Refer to How can I read the whole file content verbatim into a (expl3) string variable? for explanation of the catcode change parts.