# Add content from an external file to the input tokens for parsing

I have a possibly very long sequence of tokens which is parsed by a macro. I would like to be able to read parts of this sequence from external files, if this is possible.

I made up an MWE for this problem. The macro \myparse parses a sequence of tokens consisting of a, b, and c. The parsing is terminated by X.

Now, I added a further token i followed by a file name. I would like to have i{extern.inc} replaced by the parsed contents of the file extern.inc.

The file extern.inc contains:

aaaabcabcccc


My MWE is:

\documentclass{article}

\def\myparse{%
\afterassignment\myhandle\let\mytoken=%
}

\def\myhandle{%
\ifx\mytoken X%
\let\next=\nextX%
\else%
\ifx\mytoken a%
\let\next=\nexta%
\else%
\ifx\mytoken b%
\let\next=\nextb%
\else%
\ifx\mytoken c%
\let\next=\nextc%
\else%
\ifx\mytoken i%
\let\next=\nexti%
\fi%
\fi%
\fi%
\fi%
\fi%
\next%
}

\def\nextX{}
\def\nexta{(A)\myparse}
\def\nextb{(B)\myparse}
\def\nextc{(C)\myparse}
\def\nexti#1{\input{#1}\myparse}

\begin{document}

\myparse abccbbaaabi{extern.inc}bcX

\bigskip
I would like to have it identical to:

\myparse abccbbaaabaaaabcabccccbcX

\end{document}


This gives:

How could the macro \nexti be modified to get the desired result that the external file is parsed resp. put into the input token sequence?

The simplest strategy is to load the file with \CatchFileDef and expand its contents after inserting a new \myparse.

\begin{filecontents*}{extern.inc}
aaaabcabcccc
\end{filecontents*}
\documentclass{article}
\usepackage{catchfile}

\def\myparse{%
\afterassignment\myhandle\let\mytoken=%
}

\def\myhandle{%
\ifx\mytoken X%
\let\next=\nextX%
\else%
\ifx\mytoken a%
\let\next=\nexta%
\else%
\ifx\mytoken b%
\let\next=\nextb%
\else%
\ifx\mytoken c%
\let\next=\nextc%
\else%
\ifx\mytoken i%
\let\next=\nexti%
\fi%
\fi%
\fi%
\fi%
\fi%
\next%
}

\def\nextX{}
\def\nexta{(A)\myparse}
\def\nextb{(B)\myparse}
\def\nextc{(C)\myparse}
\def\nexti#1{\CatchFileDef\temp{#1}{\endlinechar=-1 }\expandafter\myparse\temp}

\begin{document}

\myparse abccbbaaabi{extern.inc}bcX

\bigskip
I would like to have it identical to:

\myparse abccbbaaabaaaabcabccccbcX

\end{document}


• That's definitly a good answer for my MWE. I wasn't aware of the catchfile package. Still, I'm struggling with applying this answer to my real problem ... something does not work here, but I will need more time to figure out the cause. – Thomas F. Sturm Sep 26 '14 at 9:47
• @ThomasF.Sturm Are spaces significant in the string to parse? – egreg Sep 26 '14 at 9:51
• Yes and no. The 'no' is for parsing where spaces should/are ignored. Since the sequence also contains LaTeX text passages, spaces should not be removed in general... Meanwhile, I found my adaption errors. As far as I can see, your answer works for my real application without flaw :-) Thank you very much for your help - I wasn't sure that is could work anyhow... – Thomas F. Sturm Sep 26 '14 at 10:10

There is simpler way without usage of catchfile. If you are using LaTeX, then replace the line \def\nexti#1{\input{#1}\myparse} in your MWE by:

\def\nexti#1{\expandafter\expandafter\expandafter \myparse \csname @@input\endcsname #1 }


If you are using plain TeX then the definition is more simple:

\def\nexti#1{\expandafter \myparse \input #1 }

• Before I asked this question, I was trying some \expandafter things myself, but without luck. Your solution works very good for my MWE! But for my real application, I have problem with the file ending which seems to introduce sometimes blank or \par tokens to \mytoken. Typically, the external file ends with a } (the parsed tokens take parameters) which seems to be the cause of the problem... – Thomas F. Sturm Sep 28 '14 at 10:16
• Now, I can recreate the problem for the MWE. Could your LaTeX solution be modified to parse \myparse abccbbaaabi{extern.inc} bcX also? Currently, the extra space after } is not ignored but causes an extra C. – Thomas F. Sturm Sep 28 '14 at 10:43
• @ThomasF.Sturm Include next (innermost) level to your \if\else\fi tree with the content: \else \expandafter\ifx\space\mytoken \let\next\myparse \fi. The reason of your problem: you say \let\mytoken=% at the line 4 of your code: this ignores the first space. But if you type i{foo} ab then you have two spaces: first one from the end of the file and second one explicitly written. – wipet Sep 28 '14 at 11:41