# Saving in macros plus \scantokens as an alternative to external files

I am generating a lot of TeX code automatically using Python scripts. I've been saving the TeX code to file, and then bringing those files into my documents at the appropriate places via \input. That works, but sometimes it results in a lot of external files, many of which are only a few lines long.

I'm thinking about saving the generated TeX code in macros. I could just bring in a big file of macro definitions in my preamble rather than many small files at various points in the document. The problem with this approach is that the TeX code can contain verbatim and similar content. I've developed an approach for saving such content in a macro and then retrieving it using \scantokens (minimal example below). Does this approach have any disadvantages compared to leaving the TeX code in individual external files? Are there cases where \scantokens will fail to replicate exactly what I would get from \input with an external file?

In terms of application, this is primarily related to my pythontex package. The package allows you to include Python code in a LaTeX document, execute it, and bring in its output. The Python code is saved to an external file during compilation. A script then executes each block of code, and saves each block's output in its own file. These files are automatically brought back into the LaTeX document at the appropriate place via \input. This allows the output of, say, a print() function in the Python code to be replaced by its output. Usually, printed content (the external files) would only be a couple to a few dozen lines, but they could be as large as the user wants.

\documentclass{article}

\makeatletter
\long\def\catcher#1{%
\def\name{#1}%
\begingroup
\endlinechar\^^J
\let\do\@makeother\dospecials
\catcher@i}

\begingroup
\catcode!=0
!catcode!\=12
!long!gdef!catcher@i#1\endcatcher^^J{%
!endgroup!expandafter!gdef!name{#1}}%
!endgroup

\def\getter#1{\expandafter\scantokens\expandafter{#1\empty}}
\makeatother

\catcher\myfile
Text, with verb \verb!#\$^_!
\begin{verbatim}
Verbatim
environment
\end{verbatim}
More text
\endcatcher

\begin{document}

\getter\myfile

\end{document}

-
It is hard to tell without knowing the application. For example you can process the main TeX file with Python and insert the code at appropriate places which you tag for example like a wiki {{code}}and by-pass all the coding part:) –  Yiannis Lazarides Feb 12 '13 at 22:35
@YiannisLazarides I've added an explanation of the application. I'm actually trying to get away from processing the main TeX file with Python--I actually have the Python in the main TeX file, with a way to execute it, and am working on a better way to get the output back into the document. –  G. Poore Feb 12 '13 at 23:01

I have discovered one significant issue that currently prevents \scantokens from being a general replacement for \input from a file. The LuaTeX implementation of \scantokens has a bug (referenced on tex.sx and tracked here) related to \newlinechar. So any document or package that needs to be compatible with LuaTeX, XeTeX, and pdfTeX should still use \input`.