# How to retain actual input newline/end-of-line (hex 0x0a) in macro definition

I am using filecontents package and want to include it in a macro in order to factorize some complex processing. The problem is that \begin{filecontents} needs be followed by nothing except spaces and an actual trailing newline which is unfortunately absorbed by the way TeX/LaTex processes the macro definition. Here is the M(not)WE :

\documentclass{article}
\usepackage{filecontents}
\thispagestyle{empty}

\newcommand{\cf}[2]{%
% next-line-compulsorily-ends-only-with-spaces-then-newline
\begin{filecontents}{prefix-#1.tex}
factored-complex-processing-of{#2}
\end{filecontents}
}
\show\cf
%
\cf{suffix1}{somefirststuff}
%
\begin{filecontents}{prefix-suffix2.tex}
complex-processing-of{somenextstuff}
\end{filecontents}

\begin{document}

some-package-macro-processing-using-suffix-number{1,2}

Equivalent to:

\input{prefix-suffix1.tex}
\input{prefix-suffix2.tex}

\end{document}


Because of the absorbed thus non-existing newline at the end of the \begin{filecontents} line inside the \newcommand{\cf}, TeX/LaTeX does not understand this command and processes the rest as text to be output, which causes the Missing \begin{document} error.

I have tried to seek wordarounds involving \def, \edef, \noexpand, \catcode and the like, but I did not find anything useful.

So is there a way to keep this actual input newline/end-of-line in the macro definition ?

• Are you aware that the filecotents-environment switches to verbatim-catcode-régime for tokenizing the body of the environment and thus expects the phrase \end{filecontents} to be tokenized under verbatim-catcode-rǵime also, which is not the case with your \cf-macro as its definition, inclusive the the body of the environment and the phrase \end{filecontents} get tokenized under normal catcode-régime? Mar 28 '19 at 9:34
• You don't need filecontents to write to a file. Do you want to write a single line? Mar 28 '19 at 9:50

\documentclass{article}
\usepackage{verbatim}
\usepackage{filecontents}
\thispagestyle{empty}

\newcommand\exchange[2]{#2#1}

% The integer-parameter \newlinechar usually has the value 10.
% This means if at the time of writing to text-file (La)TeX encounters a
% character whose code-point's number in (La)TeX's internal character-encoding
% (which either is ASCII or is UTF-8)  is 10, it will not write that character
% but will write a linebreak. 10 denotes the Line Feed-character and using
% (La)TeX's ^^-notation you can also write it as ^^J.

\begingroup
\newcommand\cf[2]{%
\endgroup
\newcommand{\cf}[2]{%
\scantokens{%
#1{prefix-##1.tex}^^J%
factored-complex-processing-of{##2}^^J%
#2%
}%
}%
}%
\expandafter\exchange\expandafter{\expandafter{\string\end{filecontents}}}%
{\expandafter\cf\expandafter{\string\begin{filecontents}}}%

\cf{suffix1}{ somefirststuff\LaTeX}%

\begin{filecontents}{prefix-suffix2.tex}
factored-complex-processing-of{ somenextstuff\LaTeX}
\end{filecontents}

\begin{document}

\noindent prefix-suffix1.tex is:

\verbatiminput{prefix-suffix1.tex}

\noindent\hrulefill\null

\noindent prefix-suffix2.tex is:

\verbatiminput{prefix-suffix2.tex}

\noindent\hrulefill\null

Do you see the subtle difference?

In prefix-suffix1.tex there is a space behind the phrase \verb|\LaTeX|.

In prefix-suffix2.tex there is no space behind the phrase \verb|\LaTeX|.

The reason is:

prefix-suffix1.tex comes from the \verb|\cf|-command.

The tokens forming the second argument of \verb|\cf| were
tokenized under normal catcode-r\'egime. Thus the phrase \verb|\LaTeX| got
tokenized as control-word-token. When \verb|\scantokens| did its simulated
unexpanded-writing-part, that control-word-token got written unexpanded
with a trailing space as \LaTeX{} always writes control-word-tokens with
a trailing space.\\
Besides this, hashes of catcode 6 will be doubled at writing time also.

If you don't like such side-effects, you need to have  \LaTeX{} read the
second argument of \verb|\cf| under verbatim-catcode-r\'egime. How to do
that is exhibited in the other example.

\noindent\hrulefill\null

\noindent\verb|\input{prefix-suffix1.tex}| yields:

\noindent\input{prefix-suffix1.tex}

\noindent\hrulefill\null

\noindent\verb|\input{prefix-suffix2.tex}| yields:

\noindent\input{prefix-suffix2.tex}

\end{document}


Be aware that \scantokens{<stuff>}, which is provided by the e-TeX-extensions, is similar to

\newtoks\myscratchtoks
\newwrite\myscratchwrite
...
\immediate\openout\myscratchwrite temp.tex\relax
\myscratchtoks{<stuff>}%
\immediate\write\myscratchwrite{\the\myscratchtoks}%
\immediate\closeout\myscratchwrite
\input temp.tex %\@@@input with LaTeX 2e.


I said "similar" because it is not exactly the same: \scantokensitself is expandable, while the code-snippet above is not.

E.g, with \scantokens you can do

\expandafter\def\expandafter\temp\scantokens{{definition text}}


When adapting the code-snippet above, would have to do:

\newtoks\myscratchtoks
\newwrite\myscratchwrite
...
\immediate\openout\myscratchwrite temp.tex\relax
\myscratchtoks{{definition text}}%
\immediate\write\myscratchwrite{\the\myscratchtoks}%
\immediate\closeout\myscratchwrite
\expandafter\def\expandafter\temp\input temp.tex %\@@@input with LaTeX 2e.


With both approaches unexpanded writing/simulated unexpanded writing of tokens to text-file takes place.

Be aware that when unexpanded-writing of tokens takes place, character-tokens of catcode 6 will be doubled, i.e., hashes # of catcode 6 will be doubled.

Be aware that when unexpanded-writing of tokens takes place, control-word-tokens will be written with a trailing space.

If you don't want these effects, you need to turn \cf into a macro which does read its stuff under verbatimized-catcode-conditions. If you do this, \cf must not get its arguments supplied via other macros but it must get them by reading and tokenizing from the tex-input-file. I.e., the same restrictions apply as, e.g., with the \verb-command.

I can offer a routine \UDcollectverbarg:

\documentclass{article}
\usepackage{verbatim}
\usepackage{filecontents}
\thispagestyle{empty}

\makeatletter
\newcommand\UD@firstofone[1]{#1}%
\newcommand\UD@firstoftwo[2]{#1}%
\newcommand\UD@secondoftwo[2]{#2}%
\newcommand\UD@exchange[2]{#2#1}%
\newcommand\UD@PassFirstToSecond[2]{#2{#1}}%
%%<-------------------- Code for \UDcollectverbarg -------------------->
%% Check whether argument is empty:
%%......................................................................
%% \UD@CheckWhetherNull{<Argument which is to be checked>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is empty>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is not empty>}%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
\newcommand\UD@CheckWhetherNull[1]{%
\romannumeral0\expandafter\UD@secondoftwo\string{\expandafter
\UD@secondoftwo\expandafter{\expandafter{\string#1}\expandafter
\UD@secondoftwo\string}\expandafter\UD@firstoftwo\expandafter{\expandafter
\UD@secondoftwo\string}\expandafter\expandafter\UD@firstoftwo{ }{}%
\UD@secondoftwo}{\expandafter\expandafter\UD@firstoftwo{ }{}\UD@firstoftwo}%
}%
%%......................................................................
\begingroup
\catcode\^^M=12 %
\UD@firstofone{%
\endgroup%
\newcommand\UDEndlreplace[2]{\romannumeral0\@UDEndlreplace{#2}#1^^M\relax{}}%
\newcommand*\@UDEndlreplace{}%
\long\def\@UDEndlreplace#1#2^^M#3\relax#4#5{%
\UD@CheckWhetherNull{#3}%
{ #5{#4#2}}{\@UDEndlreplace{#1}#3\relax{#4#2#1}{#5}}%
}%
}%
\newcommand\UDcollectverbarg[3]{%
\begingroup
\let\do\@makeother % <- this and the next line switch to
\dospecials        %    verbatim-category-code-régime.
\catcode\{=1      % <- give opening curly brace the usual catcode so a
%    curly-brace-balanced argument can be gathered in
%    case of the first thing of the verbatimized-argument
%    being a curly opening brace.
\catcode\ =10     % <- give space and horizontal tab the usual catcode so \UD@collectverbarg
\catcode\^^I=10   %    cannot catch a space or a horizontal tab as its 4th undelimited argument.
%    (Its 4th undelimited argument denotes the verbatim-
%     syntax-delimiter in case of not gathering a
%     curly-brace-nested argument.)
\kernel@ifnextchar\bgroup
{% seems a curly-brace-nested argument is to be caught:
\catcode\}=2    % <- give closing curly brace the usual catcode also.
\UD@collectverbarg{#1}{#2}{#3}{}%
}{% seems an argument with verbatim-syntax-delimiter is to be caught:
\do\{% <- give opening curly brace the verbatim-catcode again.
\UD@collectverbarg{#1}{#2}{#3}%
}%
}%
\newcommand\UD@collectverbarg[4]{%
\do\ %   <- Now that \UD@collectverbarg has the delimiter or
\do\^^I%    emptiness in its 4th arg, give space and horizontal tab
%    the verbatim-catcode again.
\do\^^M% <- Give the carriage-return-character the verbatim-catcode.
\long\def\@tempb##1#4{%
%\edef\@tempb{##1}%
\def\@tempb{##1}%
\@onelevel@sanitize\@tempb % <- Turn characters into their "12/other"-pendants.
%    This may be important with things like the
%    inputenc-package which may make characters
%    active/which give them catcode 13(active).
\expandafter\UDEndlreplace\expandafter{\@tempb}{#1}{\def\@tempb}% <- this starts
%    the loop for replacing endline-characters.
\expandafter\UD@@collectverbarg\expandafter{\@tempb}{#2}{#3}% <- this "spits
%    out the result.
}%
\@tempb
}%
\newcommand\UD@@collectverbarg[3]{%
\endgroup
#2{#3{#1}}%
}%
%%<---------------- End of code for \UDcollectverbarg ----------------->

% The integer-parameter \newlinechar usually has the value 10.
% This means if at the time of writing to text-file (La)TeX encounters a
% character whose code-point's number in (La)TeX's internal character-encoding
% (which either is ASCII or is UTF-8)  is 10, it will not write that character
% but will write a linebreak. 10 denotes the Line Feed-character and using
% (La)TeX's ^^-notation you can also write it as ^^J.
\newcommand{\cf}[1]{%
\UDcollectverbarg{^^J}{\UD@firstofone}{\innercf{#1}}%
}%
\begingroup
\newcommand\innercf[2]{%
\endgroup
\newcommand\innercf[2]{%
\scantokens{%
#1{prefix-##1.tex}^^J%
factored-complex-processing-of{##2}^^J%
#2%
}%
}%
}%
\UDcollectverbarg{^^J}{%
\UDcollectverbarg{^^J}{\UD@firstofone}%
}{\innercf}|\begin{filecontents}||\end{filecontents}|%

\cf{suffix1}{ somefirststuff\string#\LaTeX}%

\begin{filecontents}{prefix-suffix2.tex}
factored-complex-processing-of{ somenextstuff\string#\LaTeX}
\end{filecontents}

\begin{document}

\noindent prefix-suffix1.tex is:

\verbatiminput{prefix-suffix1.tex}

\noindent\hrulefill\null

\noindent prefix-suffix2.tex is:

\verbatiminput{prefix-suffix2.tex}

\noindent\hrulefill\null

\noindent\verb|\input{prefix-suffix1.tex}| yields:

\noindent\input{prefix-suffix1.tex}

\noindent\hrulefill\null

\noindent\verb|\input{prefix-suffix2.tex}| yields:

\noindent\input{prefix-suffix2.tex}

\end{document}


If you want the stuff in \cfs second argument to be fully evaluated/expanded before writing it to file, you can probably apply \protected@edef and \@onelevel@sanitize:

\documentclass{article}
\usepackage{verbatim}
\usepackage{filecontents}
\thispagestyle{empty}

\newcommand\exchange[2]{#2#1}

% The integer-parameter \newlinechar usually has the value 10.
% This means if at the time of writing to text-file (La)TeX encounters a
% character whose code-point's number in (La)TeX's internal character-encoding
% (which either is ASCII or is UTF-8)  is 10, it will not write that character
% but will write a linebreak. 10 denotes the Line Feed-character and using
% (La)TeX's ^^-notation you can also write it as ^^J.

\newenvironment{localscope}{}{}%

\begingroup
\makeatletter
\newcommand\cf[2]{%
\endgroup
\newcommand{\cf}[2]{%
\begin{localscope}%
\protected@edef\UD@tempa{##2}%
\@onelevel@sanitize\UD@tempa
\expandafter\exchange\expandafter{%
\expandafter\scantokens\expandafter{%
\romannumeral0\expandafter\exchange\expandafter{\expandafter{\UD@tempa}}{ %
#1{prefix-##1.tex}^^J%
factored-complex-processing-of}^^J%
#2%
}}%
{\end{localscope}}%
}%
}%
\expandafter\exchange\expandafter{\expandafter{\string\end{filecontents}}}%
{\expandafter\cf\expandafter{\string\begin{filecontents}}}%

\newcommand\firststage{\secondstage}
\newcommand\secondstage{\thirdstage}
\newcommand\thirdstage{Expanded. \LaTeX}

\cf{suffix1}{ somefirststuff \firststage}%

\begin{filecontents}{prefix-suffix2.tex}
factored-complex-processing-of{ somenextstuff \firststage}
\end{filecontents}

\begin{document}

\noindent prefix-suffix1.tex is:

\verbatiminput{prefix-suffix1.tex}

\noindent\hrulefill\null

\noindent prefix-suffix2.tex is:

\verbatiminput{prefix-suffix2.tex}

\noindent\hrulefill\null

\noindent\verb|\input{prefix-suffix1.tex}| yields:

\noindent\input{prefix-suffix1.tex}

\noindent\hrulefill\null

\noindent\verb|\input{prefix-suffix2.tex}| yields:

\noindent\input{prefix-suffix2.tex}

\end{document}
`