# How to handle verbatim material in LuaLaTeX?

While trying to implement an own working alternative to shdoc (one of its issues see here) I encountered an interesting behaviour of LuaLaTeX I cannot explain.

Consider the following code:

\documentclass{scrreprt}
\usepackage{xparse}
\usepackage{luacode}

\DeclareDocumentCommand{\terminaloutput}{+v}{
\directlua{tex.print("\luatexluaescapestring{#1}")}
}

\begin{document}
\terminaloutput{
total 10244
drwxr-xr-x 47 User Users   16384 Mar 31 18:26 .
dr-xr-xr-x  8 User Users    4096 Jun 27  2015 ..
drwxr-xr-x  9 User Users    4096 Dec 23 12:49 .editor
-rw-r--r--  1 User Users    2950 Mar 31 09:01 .bash_history
}
\end{document}


I just defined a function using a verbatim argument (\terminaloutput) which has to be verbatim since there is a _ in .bash_history. My question is why LaTeX is able to print the whole content of #1 (as expected in a continuous way) and Lua doesn't print anything.

• I get no output at all with LuaLaTeX – egreg Apr 2 '17 at 12:28
• @egreg Deleted aux files and you're right. But do you get the full output if using #1 instead of the \directlua line? – TeXnician Apr 2 '17 at 12:30
• Yes, without line breaks as expected – egreg Apr 2 '17 at 12:34
• @egreg I adapted the question. Primarily I search for the reason for the different behavior. – TeXnician Apr 2 '17 at 12:36
• An obvious problem is .. that Lua interprets in the wrong way as far as TeXing is concerned. – egreg Apr 2 '17 at 12:46

Rather than re-invent the wheel, I prefer to refer to and build on Paul Isambert's wonderful article LuaTEX: What it takes to make a paragraph, published in 2011 in Volume 32 of TUGBoat.

In this article, the author discusses inter alia how one can make use of the LuaTeX's process_input_buffer callback to handle verbatim material in a very robust and flexible manner. To quote from the article:

One of the most arcane areas of TeX is catcode management. This becomes most important when one wants to print verbatim text, i.e. code that TeX should read as characters to be typeset only, with no special characters, and things turn definitely dirty when one wants to typeset a piece of code and execute it too (one generally has to use an external file). With the process_input_buffer callback, those limitations vanish: the lines we would normally pass to TeX can be stored and used in various ways afterward.

The MWE shown below adapts Paul's TUGBoat article code to LaTeX coding practices. (Paul's code is distinctly PlainTEX-ish.) If you're at all interested in this issue, you should absolutely read Paul's original article; the passage on handling verbatim material is on pages 2 and 3 of the pdf file version of the article (pp. 69 and 70 in original published journal volume). Paul explains the issues far better than I possibly could.

• The Lua-side code consists of defining a Lua table (which will hold the verbatim material) and 3 Lua functions that do most of the work.

• The TeX-side code sets up several TeX macros that activate, invoke, and suspend the operation of the Lua functions. The main LaTeX macros are \printverbatim (you should be able to guess what it does...) and \useverbatim; the latter executes whatever is in the most recently read verbatim chunk.

• A verbatim chunk is anything sandwiched between lines that say \Verbatim and \EndVerbatim, respectively. If there's any material on the lines that contain the strings \Verbatim and \EndVerbatim, respectively, it will be ignored.

Note that I have not defined a LaTeX environment called Verbatim, although it wouldn't be too hard to do so.

\documentclass{scrreprt}
%% Set up a Lua table and 3 Lua functions
\directlua{%

verb_table = {}
function store_lines (str)
if string.find (str , "\noexpand\\EndVerbatim" ) then
luatexbase.remove_from_callback (
"process_input_buffer" , "store_lines")
else
table.insert(verb_table, str)
end
return ""
end
function register_verbatim ()
verb_table = {}
"process_input_buffer" , store_lines , "store_lines")
end
function print_lines ( catcode )
if catcode then
tex.print ( catcode , verb_table)
else
tex.print ( verb_table )
end
end

}
%% TeX-side code: define several macros
\def\Verbatim{\directlua{register_verbatim()}}
\def\useverbatim{\directlua{print_lines()}}
\def\printverbatim{%
\par
\bgroup
\setlength{\parindent}{0pt}
\ttfamily
\directlua{print_lines(1)}
\egroup
}
\def\createcatcodes{%
\bgroup
\catcode\\=12 \catcode\{=12 \catcode\}=12
\catcode\\$=12 \catcode\&=12 \catcode\^^M=13
\catcode\#=12 \catcode\^=12 \catcode\_=12
\catcode\ =13 \catcode\~=12 \catcode\%=12
\savecatcodetable 1 % '\savecatcodetable' is a LuaTeX primitive
\egroup}
\createcatcodes
\def\Space{ }
\bgroup
\catcode\^^M=13\gdef^^M{\quitvmode\par}%
\catcode\ = 13\gdef {\quitvmode\Space}%
\egroup

\begin{document}

\Verbatim
total 10244
drwxr-xr-x 47 User Users   16384 Mar 31 18:26 .
dr-xr-xr-x  8 User Users    4096 Jun 27  2015 ..
drwxr-xr-x  9 User Users    4096 Dec 23 12:49 .editor
-rw-r--r--  1 User Users    2950 Mar 31 09:01 .bash_history
\EndVerbatim
\printverbatim

\bigskip
\Verbatim
\def\lululu{%
Lua\kern-.01em
\TeX
}%
\EndVerbatim
\printverbatim

\bigskip
\useverbatim % this "registers" the macro "\lululu"
\noindent\lululu
\end{document}

• I really like this solution, but is it possible to wrap that into one macro so that the \Verbatim and \EndVerbatim could be in one call like the \terminaloutput in my original code example? – TeXnician Apr 2 '17 at 15:43
• @TeXnician - The main issue here is that one has to provide the software something specific and unmistakable to "look for" as the terminator of the verbatim material. (In the answer given above, the terminating marker is the string \EndVerbatim.) Since verbatim material can consist of fairly arbitrary material, including instances of unmatched { and }, "counting" the instances of { and } and looking for the instance of "}" that "corresponds to" or "matches" the "{" in \terminaloutput{ won't do. By the way, that's one of the shortcomings of your idea for \terminaloutput{...}. – Mico Apr 2 '17 at 16:34
• Thanks for the help. I will rethink the concept of my output lines. But for longer outputs an environment (or similar concept like above) is a better option either way. – TeXnician Apr 2 '17 at 16:35
• Just out of curiosity: There are well-debugged and reliable packages, such as fancyvrb, out there which work well with both pdfLaTeX and LuaLaTeX. Is there something the fancyvrb package doesn't do for you? – Mico Apr 2 '17 at 16:37
• I'll do that within the next days. – TeXnician Apr 4 '17 at 5:24

Mico's answer is very good and shows how a low-level implementation of verbatim buffers in LuaTeX could look like.

ConTeXt comes with this feature included and it is super easy to use.

\starttext

\startbuffer[terminal]
total 10244
drwxr-xr-x 47 User Users   16384 Mar 31 18:26 .
dr-xr-xr-x  8 User Users    4096 Jun 27  2015 ..
drwxr-xr-x  9 User Users    4096 Dec 23 12:49 .editor
-rw-r--r--  1 User Users    2950 Mar 31 09:01 .bash_history
\stopbuffer

\typebuffer[terminal]

\startbuffer[logo]
\def\lululu{%
Lua\kern-.01em
\TeX
}%
\stopbuffer

\typebuffer[logo]

\getbuffer[logo]
\lululu

\stoptext


The output is similar to Mico's answer.

• Although I do not use Context it is nice to know. – TeXnician Apr 3 '17 at 4:24

An alternative, that tries to imitate ConTeXt buffers is the scontents package. The output obtained is the same as in Mico's response, of course it is not "bullet proof", but, allow me to customize the verbatim output using fancyvrb and tcolorbox as I read in your comments.

The main function of the package is to store content, but try to imitate \typebuffer using verbatim, anyway, here's my answer:

\documentclass{article}
\usepackage{scontents}
\setlength{\parindent}{0pt}
\pagestyle{empty}
\begin{document}
\section{default}
% stored in terminal
\begin{scontents}[store-env=terminal]
total 10244
drwxr-xr-x 47 User Users   16384 Mar 31 18:26 .
dr-xr-xr-x  8 User Users    4096 Jun 27  2015 ..
drwxr-xr-x  9 User Users    4096 Dec 23 12:49 .editor
-rw-r--r--  1 User Users    2950 Mar 31 09:01 .bash_history
\end{scontents}
\typestored[1]{terminal}

% stored in logo
\begin{scontents}[store-env=logo]
\def\lululu{
Lua\kern-.01em
\TeX
}
\end{scontents}
\typestored[1]{logo}
\getstored[1]{logo}
\lululu
\end{document}


Customizing the output of \typestoredusing fancyvrb and tcolorbox:

\documentclass{article}
\usepackage{scontents}
\makeatletter
\let\verbatimsc\@undefined
\let\endverbatimsc\@undefined
\makeatother
\usepackage{fancyvrb,tcolorbox}
\newenvironment{verbatimsc} %
{\VerbatimEnvironment
\begin{tcolorbox}[colback=gray!25, boxsep=0pt, arc=0pt, boxrule=0pt]
\begin{Verbatim}[fontsize=\small]} %
{\end{Verbatim} %
\end{tcolorbox}}
\setlength{\parindent}{0pt}
\pagestyle{empty}
\begin{document}
\section{Using tcolorbox and fancyvrb}
% stored in terminal
\begin{scontents}[store-env=terminal]
total 10244
drwxr-xr-x 47 User Users   16384 Mar 31 18:26 .
dr-xr-xr-x  8 User Users    4096 Jun 27  2015 ..
drwxr-xr-x  9 User Users    4096 Dec 23 12:49 .editor
-rw-r--r--  1 User Users    2950 Mar 31 09:01 .bash_history
\end{scontents}
\typestored[1]{terminal}

% stored in logo
\begin{scontents}[store-env=logo]
\def\lululu{
Lua\kern-.01em
\TeX
}
\end{scontents}
\typestored[1]{logo}
\getstored[1]{logo}
\lululu

\end{document}