LaTeX parser to perform operations on tags (commands) and its content?

Question

Is there a LaTeX parser available that can be run from a (*nix) command line, and that allows operations on LaTeX tags/commands and its content?

What I am trying to do is, for example:

• script-wise extracting certain text blocks, like all \question{} tags within the \begin{document} ... \end{document},
• or renaming certain tags (like \newtext{blabla} to \oldtext{blabla}) [hmm, bad example since very easy to do],
• or deleting specific tags and its content, like \oldtext{}
• or just deleting the tag but retaining the content.

The main reason for not using find&replace from a LaTeX editor, or utilities like grep is problems with nested tags and closing braces on new lines.

Answer 1

Since the OP is asking quite a general question, I have worked on a solution that would be as general as possible.

EDIT: I am rewriting this post completely, because I have written another chunck of code with a completely different approach, and this new code should be much more flexible eventually (once I understand it).

It is much easier to write to a file the argument of some commands than the text that is outside. And one normally has to choose between

• reading the file, and somehow expanding the macros that we wish to omit: in other words, work in TeX's mouth. The main drawback is that we are (essentially) not allowed to use counters and define macros while the expansion is happening,

• letting TeX typeset, and store the relevant pieces as we go: this requires the text that we want to extract to be an argument of some macros

If we want to do something in between, such as leaving all the text untouched until the \begin{document}, while doing something clever with some of the content in between (e.g., also execute the \newcommands, or load some of the packages), then we are in trouble: we might want to do non-expandable things, while stil wanting to write to our output the full text (i.e. not only arguments of a command).

My (second) solution is thus essentially to make all characters active, and get them to expand to either "write <char> to a file" or just "<char>", depending on whether we are currently expanding or typesetting. A way to think about it is that we are (vaguely) emulating the typesetting process of TeX to a dvi, but instead of outputting to dvi, we output to a text file.

To use it, you can download both modify.sty and modify-2.tex (and xparse if you don't have it). Then, replace infile.tex and outfile.tex by the relevant files, and run the following minimal example.

\documentclass{minimal}
\input{modify-2.tex}
\begin{document}

\ExplSyntaxOn
\ModifyInput{infile.tex}
\tl_set:Nn \g_modify_actions_tl {
  \Modify_Extract:Nn \frac {2}                             %*
  \Modify_Extract:Nn \footnote {1}                         %*
  \Modify_DIY:Nnnnn \emph {1} { \stepcounter{mycounter} }{ %*
    \string\emph{\arabic{mycounter}. #1}     }{}
}
\ModifyDoWhatever{outfile.tex}

\end{document}

Currently, the only user commands are \Modify_Extract:Nn and \Modify_DIY:Nnnnn.

• \Modify_Extract:Nn \macro {k} defines \macro as a macro with k arguments, and requires that it be extracted from the file.

• \Modify_DIY:Nnnnn \macro {k} {<token list 1>} {<token list 2>} {<token list 3>} defines \macro to take k arguments. When it is called, the macro executes <token list 1>, then "typesets" <token list 2> (has to be expandable) to a file, and finally, it executes <token list 3> (not sure that this is useful). For instance, the \emph will now be numbered (ok, this is stupid, but think of knowing the text of the last footnote preceeding the emphasized text that we are looking at).

For your specific problem, I would do

\documentclass{minimal}
\input{modify-2.tex}
\begin{document}

\ExplSyntaxOn
\ModifyInput{infile.tex}
\tl_set:Nn \g_modify_actions_tl {
  \Modify_Extract:Nn \question {1}   
}
\ModifyDoWhatever{outfile.tex}

\end{document}

or something more fancy using \Modify_DIY:Nnnnn.

In fact, I lied a bit: I do not play around with catcodes too much: instead, I replace each character by a control sequence that represents it. This forces me to implement variants of \newcommand with arguments delimited in the right way. Another technical piece is the treatment of spaces.

Also, I have not implemented optional arguments. And nesting is currently failing. I believe that this can be fixed by being less sloppy in what is expanded or not.

Answer 2

I would redefine the commands/environments to do what you want and then call the old definitions. For example, let us say you want all \question{} content in some file (same filename and extension "loq").

\AtBeginDocument{
  \newwrite\myOutputStream
  \immediate\openout\myOutputStream=\jobname.loq
}
\AtEndDocument{
  \immediate\closeout\myOutputStream
}
\let\oldQuestion\question
\renewcommand\question[1]{\immediate\write\myOutputStream{#1}\noexpand\oldQuestion{#1}} %edited here

The code above has not been tested, but it should give you an idea of how to proceed, at least to redefine commands. Note the use of let to save the old "program" rather than its "outcome".

---

EDIT: code was updated

Answer 3

Sed and Awk are your friends. They should be powerful enough to deal with nested tags and so on. (I don't know for sure though)

An alternative would be to take a look at TeXcount and see how that parses things, see if that helps...

Answer 4

I would probably go for perl, where you can use regular expressions to match balanced text or use modules such as Text::Balanced.

But maybe that's just me.