\CatchFileDef vs ( \input or \@@input )

Question

I discovered catchfile here and now I would like to understand how works this package. Below some tests with \foreach or \@for

\documentclass{article}
\usepackage{filecontents,catchfile,pgffor}

\begin{document}

\def\mylist{1,2,3,4} 
Test 1:\foreach \i in \mylist {\hspace{\i cm}\i} 

\begin{filecontents*}{mylist.dat}
1,2,3,4
\end{filecontents*}

\CatchFileDef{\tempa}{mylist.dat}{}% 

Test 2:\foreach \i in \tempa{\hspace{\i cm}\i} 

\makeatletter 

Test 3:\@for\i:=\tempa\do{\hspace{\i cm}\i}

\def\tempb{\@@input mylist.dat } 

% I know this is not the right way but I always find it difficult to explain


Test 4: this is the content : \tempb  
% \foreach \i in \tempb 
% {\hspace{\i cm}\i}    %  

Test 5:\@for\i:=\tempb\do{\hspace{\i cm}\i} 

\begin{filecontents*}{myfile.dat}
\def\tempc{1,2,3,4}
\end{filecontents*}  
\input{myfile.dat}

% last try ! and it works
Test 6:\foreach \i in \tempc{\hspace{\i cm}\i}   

\end{document}

A) Test 4 does not work. but why Test 2 with \CatchFileDef works ? It would be interesting to have a clean explanation about this kind of problem !

B) Now why Test 6 works ?

Answer 1

The definition of \CatchFileEdef might be simplified into

\newcommand\easyCatchFileEdef[3]{%
   \begingroup\everyeof{\noexpand}%
   #3\relax
   \xdef\@temp{\@@input #2\space}%
   \endgroup
   \let#1\@temp
}

but Heiko Oberdiek adds to this many checks. The \everyeof{\noexpand} is necessary to neutralize the implicit empty line that TeX adds to every file it inputs. The \@@input command is the primitive \input, which is expandable (with null expansion); the final \space is there for coping with the conventions of TeX regarding \input (primitive): a space is looked for and ignored. The third argument is used to add settings when inputting the file, for instance to \endlinedchar.

With \xdef everything is expanded (it's \global\edef), but then #1 is defined only locally.

More complicated is \CatchFileDef, because one wants to expand only \@@input and not what is actually input. The unabridged version is

\makeatletter
\def\easyCatchFileDef#1#2#3{%
  \begingroup\everyeof\expandafter{\CatchFileEOF\expandafter\CatchFileFinish\noexpand}%
  \expandafter\long\expandafter\def\expandafter\CatchFileDo
    \expandafter##\expandafter1\CatchFileEOF{\edef\CatchFileFinish{\endgroup
      \unexpanded{\edef#1{\unexpanded{##1}}}}}%
  #3\relax
  \expandafter\CatchFileDo\@@input #2\relax}
\begingroup
  \lccode65=64 % lowercase('A') = '@'
  \lccode66=64 % lowercase('B') = '@'
  \catcode65=8 % catcode('A') = subscript
  \catcode66=3 % catcode('B') = math shift
\lowercase{\endgroup
  \def\CatchFileEOF{AB}%
}
\makeatother

The tokens \CatchFileEOF are two @, the first with catcode 8 and the second with catcode 3 (that nobody is supposed to have in a file) and, essentially, what is executed with \easyCatchFiledef{\my}{file}{} is

\def\CatchFileDo#1@@{%
  \edef\CatchFileFinish{\endgroup
    \unexpanded{\edef\my{\unexpanded{#1}}}}}%
\CatchFileDo<contents of file.tex>@@\CatchFileFinish

because the \expandafter before \CatchfileDo will expand \@@input and present TeX with the contents of file.tex. In the lines above @@ represent the \CatchFileEOF special delimiter tokens.

Actually catchfile.sty contains code for the case when \unexpanded is not what is in the present version of e-TeX, but this is irrelevant for the discussion. A clever job by Heiko, as usual.

Answer 2

The \CatchFileDef macro works AFAIK with some e-TeX trickery. Without that you get an error if you use TeX's \input (renamed to \@@input within LaTeX) inside an expandable context like \edef. In this case an error like "End of file detected while ..." is triggered.

Your test 4 doesn't work because you are using \@@input in a simple \def here:

\def\tempb{\@@input mylist.dat } 
\foreach \i in \tempb {..}

Note that \tempb does not contain the content of the file but rather the \@@input macro with the filename, which is not good for \foreach. It's like using \foreach \i in {\@@input mylist.dat} {..}, which doesn't make much sense. You would need to use \edef instead to expand the \@@input before so that \tempb does contain the file content first. However, this does not work as mentioned above.

Test 6 works as because you define the macro \tempc with the comma-separated content directly, just inside an input-file, which doesn't make any difference here. Does test 5 work for you? It would be more surprising for me.