Simple pre-parser to separate TeX and comments?

Question

I want to write a mathematical paper both with formulas and with program snippets, side by side. However I do not want the programs displayed or documented by TeX, just be able to extract them into a file if desired.

Note that my intention has nothing to do with literal programming but perhaps I can (mis-) use some of these tools for my purpose? For example assuming a special token %# for the pre-parser such a file could look like:

\documentclass[]{article}
\begin{document}
The factorial is an important function:
\begin{equation}
    n! = \prod_{k=1}^n k
\end{equation}
%#  n := 1;
%#  for k from 1 to n 
%#      n := n * k;
\end{document}

My question: Are there tools (or editors) which can preparse a TeX document and split it into two (or more) different files intended for further processing by different tools? Or any other ideas using TeX features?

EDIT: In extension to the excellent answer of jfbu: Assuming that the code snippet is immediate after a numbered equation, could the number of the equation also we written out to the code file? This would amount to a cross-referencing between the two outputs. (If this is not possible, how can the internal snippet counter introduced in the 'elaborated answer' of jfbu be written out (with a prepended comment sign like # or // ?).

Answer 1

You can do this from inside tex, without external tools.

This update generates one file for each code snippet with automatic (customizable) numbering. This example produces filename-code-01.py, filename-code-02.py, filename-code-03.py, filename-code-04.py corresponding to the four code snippets (two are in the preamble).

Additional update in response to an edit to the OP: the first line of each output code snippet is now a comment line with the number of the code snippet. The question about using the equation number is more delicate, as the extraction of the code snippets is done as part or the preamble code, before anything has been actually typeset in the document.

$ ls preparseC*
preparseC-code-01.py    preparseC-code-04.py    preparseC.log
preparseC-code-02.py    preparseC.aux       preparseC.tex
preparseC-code-03.py    preparseC.dvi

The contents of preparseC-code-01.py:

# Code snippet 1
n := 1;
for k from 1 to n
    n := n * k;

The line tag used to identify code snippets in the .tex source file is %#<space><space>.

Code:

\documentclass{article}

\newcounter{snippetno}

% customize \thesnippetno as desired, for example this produces
% filename-code-01.py
% filename-code-02.py
% ...
% filename-code-10.py
% etc...
% This command should be expandable
\renewcommand{\thesnippetno}
  {\jobname-code-\ifnum\value{snippetno}<10 %<- leave a space
        0\fi
   \arabic{snippetno}.py}


%%%%%%%% PREPARSING
\newread\parsein
\openin\parsein \jobname.tex

\newwrite\parseout
% this version will create one file for each code snippet
\newif\ifOutputtingLines

% adapt the following to the line tag you want to use
% the \detokenize is not needed here, but in case your tag
% uses letters, put them in it (do not use \catcode for letters
% a they may be in use in \def \endgroup etc..)
%
% THIS VERSION USES %#<space><space> AS LINE TAG
% (two spaces must be present and will be removed in the outputs)
\begingroup
%% ADDED DEFINITION OF \COMMENTTAG FOR USE IN FIRST LINE OF CODE SNIPPET FILES
\catcode`\% 12
\catcode`\# 12
\def\x{\endgroup\def\COMMENTTAG{#}\edef\LineTag{\detokenize{%#}\space\space}}
\x
%\show\LineTag % debugging

\begingroup
\edef\x{\endgroup
        \unexpanded{\def\CheckLineAux #1}\LineTag\relax \unexpanded{{#1}}
        \unexpanded{\def\CheckLine #1}\LineTag \unexpanded{#2}\relax
        \unexpanded{{\if\relax #1\relax 
                        \ifOutputtingLines\else
           \stepcounter{snippetno}%
           \immediate\openout\parseout \thesnippetno\relax
%% ------------------------ ADDED TO INSERT CODE SNIPPET NUMBER IN THE FILE
           \immediate\write\parseout 
                  {\COMMENTTAG\space Code snippet \arabic{snippetno}}%
%% ------------------------
           \OutputtingLinestrue
                        \fi
           \immediate\write\parseout {\CheckLineAux #2\relax}%
                     \else
                        \ifOutputtingLines
                          \immediate\closeout\parseout 
                          \OutputtingLinesfalse
                        \fi
                     \fi}}%
}
\x

\begingroup\endlinechar-1
\loop
  \ifeof\parsein
    % if \end{document} is not missing no need to \closeout\parseout
    % necessarily already done, and OutputtingLines toggle necessarily false
    \closein\parsein
  \else
    \readline\parsein to \tmpline
    \if\relax\tmpline\relax % found empty line
        \ifOutputtingLines\immediate\closeout\parseout
              \OutputtingLinesfalse
        \fi
    \else
        \expandafter\expandafter\expandafter \CheckLine
                \expandafter \tmpline\LineTag \relax
    \fi 
\repeat
\endgroup

%%%%%%%% END OF PREPARSING

% Some code snippets may already be put here in the preamble
% 
%#  n := 1;
%#  for k from 1 to n 
%#      n := n * k;

%#  y := 1;
%#  for k from 1 to n 
%#      y := y * (x + k -1);

% Notice that in this variant the line tag is %#<space><space>
% and is removed on output

\begin{document}
The factorial is an important function:
\[
    n! = \prod_{k=1}^n k
\]
%#  n := 1;
%#  for k from 1 to n 
%#      n := n * k;
The (so-called) Pochhammer coefficient also:
\[
    (x)_n = \prod_{k=1}^n (x+k-1)
\]
%#  y := 1;
%#  for k from 1 to n 
%#      y := y * (x + k -1);
\end{document}

---

This is the first version of the answer: (I have removed a \makeatletter which was useless, and commented out the \show\LineTag debugging line)

The following will on compilation also extract to filename-code the tagged lines.

  n := 1;
  for k from 1 to n
      n := n * k;

Code:

\documentclass{article}

%%%%%%%% PREPARSING
\newread\parsein
\openin\parsein \jobname.tex
\newwrite\parseout
\immediate\openout\parseout \jobname-code

% adapt the following to the line tag you want to use
% the \detokenize is not needed here, but in case your tag
% uses letters, put them in it (do not use \catcode for letters
% a they may be in use in \def \endgroup etc..)
\begingroup
\catcode`\% 12
\catcode`\# 12
\def\x{\endgroup\edef\LineTag{\detokenize{%#}}}
\x
%\show\LineTag % debugging
\begingroup
\edef\x{\endgroup
        \unexpanded{\def\CheckLineAux #1}\LineTag\relax \unexpanded{{#1}}
        \unexpanded{\def\CheckLine #1}\LineTag \unexpanded{#2}\relax
        \unexpanded{{\if\relax #1\relax 
                      \immediate\write\parseout {\CheckLineAux #2\relax}%
                     \fi}}%
}
\x

\begingroup\endlinechar-1
\loop
  \ifeof\parsein
    \immediate\closeout\parseout
    \closein\parsein
  \else
    \readline\parsein to \tmpline
    \if\relax\tmpline\relax\else
        \expandafter\expandafter\expandafter \CheckLine
                \expandafter \tmpline\LineTag \relax
    \fi 
\repeat
\endgroup

%%%%%%%% END OF PREPARSING

\begin{document}
The factorial is an important function:
\[
    n! = \prod_{k=1}^n k
\]
%#  n := 1;
%#  for k from 1 to n 
%#      n := n * k;
\end{document}

Answer 2

There are more extensive tools but if you save your example in fff.tex then grep and sed are all you need

grep -v "^%#" fff.tex

produces

\documentclass[]{article}
\begin{document}
The factorial is an important function:
\[
    n! = \prod_{k=1}^n k
\]
\end{document}

and

grep "^%#" fff.tex | sed "s/^%#//"

produces

  n := 1;
  for k from 1 to n 
      n := n * k;

Answer 3

It can also be done with a simple perl script, which for me is more readable than grep and sed with options.

#!/usr/bin/perl

while(<STDIN>)
{
  if( (substr $_, 0, 2) eq "%#" )
    {
      print substr $_, 2;
    }
}

Document saved in document.tex and perl code saved in extractComments.pl.

Produces:

hpek@melda:~/programming/perl$ cat document.tex |./extractComments.pl 
  n := 1;
  for k from 1 to n 
      n := n * k;
hpek@melda:~/programming/perl$ 

Answer 4

My thoughts: If your code chunk is R (http://www.r-project.org/), I would consider knitr, http://yihui.name/knitr/ (or now rather old Sweave, http://www.stat.uni-muenchen.de/~leisch/Sweave/).

If it is not, I would try the extract package, http://ctan.org/pkg/extract, if it fits your needs.

Personally, I would probably go for the listings package, http://mirrors.nic.cz/tex-archive/macros/latex/contrib/listings/listings.pdf, the codes can be typeset right away, and, if needed, we could turn on/off its environments (then they would behave as comments), extract them on-the-fly and we could have code chunks inside and outside the main TeX file compilable by other programs at the same time.