# LaTeX parser to perform operations on tags (commands) and its content?

Is there a LaTeX parser available that can be run from a (*nix) command line, and that allows operations on LaTeX tags/commands and its content?

What I am trying to do is, for example:

• script-wise extracting certain text blocks, like all \question{} tags within the \begin{document} ... \end{document},
• or renaming certain tags (like \newtext{blabla} to \oldtext{blabla}) [hmm, bad example since very easy to do],
• or deleting specific tags and its content, like \oldtext{}
• or just deleting the tag but retaining the content.

The main reason for not using find&replace from a LaTeX editor, or utilities like grep is problems with nested tags and closing braces on new lines.

-

Since the OP is asking quite a general question, I have worked on a solution that would be as general as possible.

EDIT: I am rewriting this post completely, because I have written another chunck of code with a completely different approach, and this new code should be much more flexible eventually (once I understand it).

It is much easier to write to a file the argument of some commands than the text that is outside. And one normally has to choose between

• reading the file, and somehow expanding the macros that we wish to omit: in other words, work in TeX's mouth. The main drawback is that we are (essentially) not allowed to use counters and define macros while the expansion is happening,

• letting TeX typeset, and store the relevant pieces as we go: this requires the text that we want to extract to be an argument of some macros

If we want to do something in between, such as leaving all the text untouched until the \begin{document}, while doing something clever with some of the content in between (e.g., also execute the \newcommands, or load some of the packages), then we are in trouble: we might want to do non-expandable things, while stil wanting to write to our output the full text (i.e. not only arguments of a command).

My (second) solution is thus essentially to make all characters active, and get them to expand to either "write <char> to a file" or just "<char>", depending on whether we are currently expanding or typesetting. A way to think about it is that we are (vaguely) emulating the typesetting process of TeX to a dvi, but instead of outputting to dvi, we output to a text file.

To use it, you can download both modify.sty and modify-2.tex (and xparse if you don't have it). Then, replace infile.tex and outfile.tex by the relevant files, and run the following minimal example.

\documentclass{minimal}
\input{modify-2.tex}
\begin{document}

\ExplSyntaxOn
\ModifyInput{infile.tex}
\tl_set:Nn \g_modify_actions_tl {
\Modify_Extract:Nn \frac {2}                             %*
\Modify_Extract:Nn \footnote {1}                         %*
\Modify_DIY:Nnnnn \emph {1} { \stepcounter{mycounter} }{ %*
\string\emph{\arabic{mycounter}. #1}     }{}
}
\ModifyDoWhatever{outfile.tex}

\end{document}


Currently, the only user commands are \Modify_Extract:Nn and \Modify_DIY:Nnnnn.

• \Modify_Extract:Nn \macro {k} defines \macro as a macro with k arguments, and requires that it be extracted from the file.

• \Modify_DIY:Nnnnn \macro {k} {<token list 1>} {<token list 2>} {<token list 3>} defines \macro to take k arguments. When it is called, the macro executes <token list 1>, then "typesets" <token list 2> (has to be expandable) to a file, and finally, it executes <token list 3> (not sure that this is useful). For instance, the \emph will now be numbered (ok, this is stupid, but think of knowing the text of the last footnote preceeding the emphasized text that we are looking at).

For your specific problem, I would do

\documentclass{minimal}
\input{modify-2.tex}
\begin{document}

\ExplSyntaxOn
\ModifyInput{infile.tex}
\tl_set:Nn \g_modify_actions_tl {
\Modify_Extract:Nn \question {1}
}
\ModifyDoWhatever{outfile.tex}

\end{document}


or something more fancy using \Modify_DIY:Nnnnn.

In fact, I lied a bit: I do not play around with catcodes too much: instead, I replace each character by a control sequence that represents it. This forces me to implement variants of \newcommand with arguments delimited in the right way. Another technical piece is the treatment of spaces.

Also, I have not implemented optional arguments. And nesting is currently failing. I believe that this can be fixed by being less sloppy in what is expanded or not.

-
Thanks for the effort. I was especially looking for extracting blocks of text, so the most useful suggestion was the extract package. However, it has a number of limitations, such as that it only allows extraction of a limited set of supported commands. MY own custom commands are not supported :-( –  Rabarberski Jan 20 '11 at 10:01
It is much easier to extract the argument of macros than the text that is outside. So maybe what you want to do is simpler than I thought. Am I right that you want to get \somecommand{text} to write \othercommand{text} to a file? If you can be more explicit, I'll try to write something this weekend. –  Bruno Le Floch Jan 20 '11 at 10:26
My question generalizes on the problem I have currently at hand, which is generating a PDF with only exercise questions from a LaTeX document which contains both the descriptions, questions and answers of the excercise notes. The questions are marked using a custom question command like so: \question{What is STP?}. So I reasoned that to do this, I have to somehow create a new .tex file with only the preamble(the part before \begin{document}), and all the \question{xxxx} blocks within the document environment. –  Rabarberski Jan 20 '11 at 11:17
BTW: I have already looked at the CTAN Exercise package, which is quite nice, but doesn't allow to print only the exercises –  Rabarberski Jan 20 '11 at 11:40
I will need similar functionalities for another package of mine. But I cannot come up with any clever user interface (I think that I can fill in the technical details). Any idea? –  Bruno Le Floch Jan 20 '11 at 11:57

I would redefine the commands/environments to do what you want and then call the old definitions. For example, let us say you want all \question{} content in some file (same filename and extension "loq").

\AtBeginDocument{
\newwrite\myOutputStream
\immediate\openout\myOutputStream=\jobname.loq
}
\AtEndDocument{
\immediate\closeout\myOutputStream
}
\let\oldQuestion\question
\renewcommand\question[1]{\immediate\write\myOutputStream{#1}\noexpand\oldQuestion{#1}} %edited here


The code above has not been tested, but it should give you an idea of how to proceed, at least to redefine commands. Note the use of let to save the old "program" rather than its "outcome".

EDIT: code was updated

-
Good starting point, but it is not completely working for me. [BTW, there are two typos in the code: \jobname.loq should be jobname.log, and myOutputStreamF on the last line should be without F] The problem is that this allows to write out the text inside \question commands, but not the both the command and the text. I've tried reinserting the command by changing myOutputStream{#1} to myOutputStream{\textbackslash question\{#1\textbackslash\}} but that generates an error. –  Rabarberski Jan 20 '11 at 10:19
@Rabarberski: I used loq on purpose for "list of questions". Not a typo. That way, you have a 'clean' file. The second is a typo (no F needed). As for the error in your change, you are defining an infinite loop (and maybe I am too). In that case, I would use a \noexpand before \oldQuestion. –  Hector Jan 22 '11 at 10:10

Sed and Awk are your friends. They should be powerful enough to deal with nested tags and so on. (I don't know for sure though)

An alternative would be to take a look at TeXcount and see how that parses things, see if that helps...

-
Sed and Awk are probably not powerful enough to deal with arbitrary nested braces. See this question on StackOverflow. But some hacks exist which might be good enough (see the comments on the top-voted answer there). –  Matthew Leingang Jan 19 '11 at 14:15
@Matthew Good point, but I expect that normally LaTeX isn't going to be nested to an extent where this is such an issue... –  Seamus Jan 21 '11 at 15:45

I would probably go for perl, where you can use regular expressions to match balanced text or use modules such as Text::Balanced.

But maybe that's just me.

-
Given that balanced parentheses is the canonical example of a nonregular language, I sure wish Perl had called it something else. –  TH. Jan 20 '11 at 13:28