# Allow macro to end without new line

The following macro, which a full MWE can be found at Unmodified Source Code ,

\def\SourceCode{%
\begingroup%
\endlinechar\^^J%
\catcode\\=12\catcode\^^M=12\catcode\#=12\catcode\~=12\catcode\%=12\catcode\^=12\catcode\_=12\catcode\@=12\catcode\ =12\catcode\|=12%
\SourceCodeAux}%
\def\SourceCodeAux#1#2{\endgroup\directlua{print("\luaescapestring{#1}")}}%


has the issue that the last terminating } cannot have anything else after it, on it's line:

\begin{itemize}
\item 2
\SourceCode{
{this
is
a
test}
}\end{itemize} % < This line is the problem, } \end{itemize} works but produces protected space in pdf.


will not work unless \end{itemize} is moved to the next line or a space is put between the } \end{itemize} (which then puts a protected space in the pdf)

The issue seems to be how the macro handles the last character but I don't quite understand what is going on. Is there any way to get the }\end{itemize} case to work?

I'm a bit confused on what the second argument for \def\SourceCodeAux#1#2 is doing. It is required but I'm not sure how it actually works. I've replaced it with other stuff and can get }\end{itemize} to work but end up with protected spaces or other characters showing up in the pdf. My thinking is if I can insert a new-line into the stream right after the final } then TeX will see }\^^M\end{itemize} and allow it to work. Anyways, maybe someone can enlighten me on what exactly is going on.

here is how I see the macro:

1. Start new group to locally define catcodes(so the catcode modifications do not effect anything but what is in the macro)

2. Modify endlinechar and catcodes so that tokens after the macro call are not "Special"

3. Call an auxiliary macro that accepts two arguments. The first being will end up being the token stream(the source code argument "passed" to the original macro call). The second argument is then going to be the last token in the source code. (In the problem case happens to be a \ which gets picked up "accidently" and breaks the \end{itemize} (tex see's it as end{itemize})

4. endgroup is called to return the catcodes back to normal then processing is done on the first "argument". (I'm not quite sure how the macro knows when to stop processing tokens though but I guess it has something to do with some trick using the #2 which is also what is causing the problem)

Here is a full MWE:

    \documentclass{book}
\usepackage{luatex}
\directlua{tex.enableprimitives('',tex.extraprimitives())}
\begin{document}

\def\SourceCode{%
\begingroup%
\endlinechar\^^J%
\catcode\\=12\catcode\^^M=12\catcode\#=12\catcode\~=12\catcode\%=12\catcode\^=12\catcode\_=12\catcode\@=12\catcode\ =12\catcode\|=12%
\SourceCodeAux}%
\def\SourceCodeAux#1#2{\endgroup\directlua{print("\luaescapestring{#1}")}}%

\begin{itemize}
\item 1
\SourceCode{
{this
is
a sucessful
test}
}
\end{itemize}

\iftrue
\begin{itemize}
\item 2
\SourceCode{
{this
is
a failed
test}
}\end{itemize}
\fi
\end{document}

-
"I'm not quite sure how the macro knows when to stop processing tokens". Note that { } are not changed, so #1 is the next argument (token or {..}). The #2 might be just there to remove the space created by a line break, i.e. the macro is designed to have a line break (or space) after it. I guess that without it the line break (i.e. EOF-character) is placed verbatim in the document, which is not what you want. I remember having basically the same trouble programming a similar macro for ydoc. Instead of #2 use a macro which checks if the following character is EOL and discards it. –  Martin Scharrer Sep 11 '12 at 6:03
Please don't use quote blocks for highlighting of your own texts. There are really just for quoting other sources. –  Martin Scharrer Sep 11 '12 at 6:04

The problem is that TeX adds the End-Of-Line (EOL, \endlinechar) character at the end of the line when it reads the full line, not when it processes the line end. This means the EOL of the last line with any code after the actual \SourceCode content is still a ^^J (prints as an uppercase Ohm). The #2 is included to gobble a following EOL, which is assumed to follow. If not, the next verbatim character is gobbled instead and the ^^J EOL is still there.

A solution is to use a cleanup macro which reads the rest of the line and removes the ^^J while reinserting the other code of the source code line. Note that the normal EOL would insert a space, which is included in the cleanup macro, but might not always behave exactly the same in all cases. Also this will break with an TeX error if there is a comment in that line which is masking the EOL character.

In general it is a bad idea to mix the first and last lines of a multi-line verbatim environment or macro with normal material, so having a line break after the } is actually a very good idea.

\documentclass{book}
\usepackage{luatex}
\directlua{tex.enableprimitives('',tex.extraprimitives())}
\begin{document}

\def\SourceCode{%
\begingroup%
\endlinechar\^^J%
\catcode\\=12\catcode\^^M=12\catcode\#=12\catcode\~=12\catcode\%=12\catcode\^=12\catcode\_=12\catcode\@=12\catcode\ =12\catcode\|=12%
\SourceCodeAux}%
\def\SourceCodeAux#1{\endgroup\directlua{print("\luaescapestring{#1}")}\SourceCodeCleanup}%

\def\SourceCodeCleanup#1^^J{#1 }%

\begin{itemize}
\item 1
\SourceCode{
{this
is
a successful
test}
}
\end{itemize}

\iftrue
\begin{itemize}
\item 2
\SourceCode{
{this
is
now also a successful
test}
}\end{itemize}
\fi
\end{document}

-

I know from the 'thread' of questions that xparse might not be what you want, but there is no problem using a verbatim argument here to get the output you indicate

\documentclass{article}
\usepackage{luatex}
\directlua{tex.enableprimitives('',tex.extraprimitives())}
\usepackage{xparse}
\NewDocumentCommand{\SourceCode}{+v}{\directlua{print("\luaescapestring{#1}")}}
\begin{document}
\begin{itemize}
\item 2
\SourceCode{
{this
is
a
test}
}\end{itemize}
\end{document}


A demonstration that this can be used to handle ^^M (end-of-line) within TeX (the above demo prints the output at the Lua level):

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{xparse}
\ExplSyntaxOn
\char_set_catcode_other:n { \^^M }
\NewDocumentCommand { \SourceCode } { +v }
{
\tl_set:Nn \l_tmpa_tl {#1}
\tl_replace_all:Nnn \l_tmpa_tl { ^^M } { \par }
\ttfamily { \l_tmpa_tl }
}
\char_set_catcode_end_line:n { \^^M }
\ExplSyntaxOff
\begin{document}
\begin{itemize}
\item 2
\SourceCode{
{this is
is
a
test}
}\end{itemize}

\end{document}


This is needed because a truly verbatim grab of each line includes ^^M at the end of each line. You can see that by showing an analysis of the grabbed tokens:

\documentclass{article}
\usepackage{xparse,l3tl-analysis}
\ExplSyntaxOn
\NewDocumentCommand { \SourceCode } { +v }
{ \tl_show_analysis:n {#1} }
\ExplSyntaxOff
\begin{document}
\begin{itemize}
\item 2
\SourceCode{
{this is
is
a
test}
}\end{itemize}

\end{document}
`
-
The issue with xparse's verbatim is that it doesn't seem to keep EOL's. The result is "thisisatest" which is unacceptable. I did modify it to change the catcodes and endlinechar similar to the original but it throws an error. I imagine xparse's "+v" processes the argument before you get a chance to modify the catcodes and such and there is no easy way to get it to work well. –  AbstractDissonance Sep 11 '12 at 17:38
@AbstractDissonance I've edited my answer to cover this case in some detail –  Joseph Wright Sep 11 '12 at 18:22
The first one updated one does print(to pdf though) the EOL's. The second one does not show any EOL's characters. In any case, how would one get it to work with lua? I need the exact same behavior of the working case using TeX's def. The whole point of the macro is to pass the argument to lua "unharmed". I've tried modifying your macro's to do this but they all fail in some way. –  AbstractDissonance Sep 11 '12 at 19:16