Accessing TeX token or exporting filtered plain text

Question

(updated according to gained information)

In the context of keyword extraction from arbitrary (La)TeX documents and simple manipulations with them (automated annotation), I would like to be able to do the following things:

1. Access a stream of tokens with information required to build tree-like representation (i.e., I need to know when environment starts, when it ends, when math formula occurs)
2. Export plain text representation of the document while being able to filter out commands (for example, remove equations or some sections).

I already have a very complicated LaTeX parser written in C++, but still I am unable to avoid the problem of expansion of commands.

Basically, if an author defines a command \keyword that just prints keyword, I cannot deal with this in a general manner. (And people really do that and tend to do this with the most important keywords!)

So, on one hand I need to get a plain text representation of the document while filtering out the math statements and not loosing information about position of the commands and environments (to aid sentence tokenizer and keyword extractor).

On the other hand, having an arbitrary TeX document I need to find a node with a specific word in it, manipulate it (remove/replace/wrap in a link) and then compile the document.

I vaguely understand that this should be possible using LuaTeX or ConTeXt, but I wasn't able to find accessible examples on LuaTeX wiki.

---

Update: I think I need to clarify that I cannot control the incoming documents, so the goal is to analyse and manipulate arbitrary valid LaTeX code.

Bounty update: Existing answer provides lots of valuable information about achieving my goals with writing input documents in ConTeXt, which is unfortunately inapplicable

Bounty update 2: To provide you the scale: we need to process more than a million documents and the number is increasing.

Answer 1

In ConTeXt, one option is to export your document as XML. You can then manipulate the resulting XML document using standard XML tools in your favourite programming language.

As an example, consider this example:

\setupbackend[export=yes]
\setupinteraction[state=start]

\starttext
\startsection[title={Test of export}]
  This is to test how \CONTEXT\ export works. First, we input a test file
  below.

  \startnarrower
    \input ward
  \stopnarrower

 Let's see some other environments, such as\footnote{Note that we define a
 float in \in{Figure}[fig:mill].}

  \startitemize[n]
    \item A table:

      \startTABLE
        \NC 1 \NC 2 \NC 3 \NC \NR
        \NC 1 \NC 2 \NC 3 \NC \NR
        \NC 1 \NC 2 \NC 3 \NC \NR
      \stopTABLE

    \item A formula

      \startformula
        α^2 + β^2 = 2
      \stopformula

    \item A figure

      \externalfigure[mill.png][width=5cm]

  \stopitemize

  \startplacefigure
      [title={This is a Mill}, reference={fig:mill}]
    \externalfigure[mill.png][width=5cm]
  \stopplacefigure

\stopsection
\stoptext

and process it using context filename. This will create a directory \jobname-export with the following structure:

\jobname-export
|
+-- images
|   +--- <image file1>
|   +---- ....
+-- style
|   +-- \jobname-defaults.css
|   +-- \jobname-images.css
|   +-- \jobname-styles.css
|   +-- \jobname-template.css
+-- \jobname-div.xhtml
+-- \jobname-pub.lua
+-- \jobname-raw.xml
+-- \jobname-tag.xhtml

Out of these, the \jobname-raw.xml contains the AST of the complete document. For the above document it is:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>

<!--

    input filename   : test
    processing date  : Tue Jun 14 19:58:26 2016
    context version  : 2016.06.02 21:28
    exporter version : 0.34

-->

<?xml-stylesheet type="text/css" href="styles/test-defaults.css" ?>
<?xml-stylesheet type="text/css" href="styles/test-images.css" ?>
<?xml-stylesheet type="text/css" href="styles/test-styles.css" ?>

<document date="Tue Jun 14 19:58:26 2016" file="test" context="2016.06.02 21:28" language="en" xmlns:m="http://www.w3.org/1998/Math/MathML" version="0.34">
 <metadata>
 </metadata>
 <section detail="section" chain="section" implicit="1" level="3">
  <sectionnumber>1</sectionnumber>
   <sectiontitle>Test of export</sectiontitle>
  <sectioncontent>
This is to test how ConTEXt export works. First, we input a test file below.
   <break/>
The Earth, as a habitat for animal life, is in old age and has a fatal illness. Several, in fact. It would be happening whether humans had ever evolved or not. But our presence is like the effect of an old-age patient who smokes many packs of cigarettes per day <subsentencesymbol symbol="middle">—</subsentencesymbol> and we humans are the cigarettes.
   <break/>
Let’s see some other environments, such as<descriptionsymbol detail="footnote" insert="1"><sup>1</sup></descriptionsymbol> 
   <itemgroup detail="itemize" chain="itemize" level="1" symbol="n">
    <item>
     <itemtag>1.</itemtag>
     <itemcontent>A table:<table><tablerow><tablecell align="flushleft"> 1 </tablecell><tablecell align="flushleft"> 2 </tablecell><tablecell align="flushleft"> 3 </tablecell></tablerow><tablerow><tablecell align="flushleft"> 1 </tablecell><tablecell align="flushleft"> 2 </tablecell><tablecell align="flushleft"> 3 </tablecell></tablerow><tablerow><tablecell align="flushleft"> 1 </tablecell><tablecell align="flushleft"> 2 </tablecell><tablecell align="flushleft"> 3 </tablecell></tablerow></table></itemcontent>
    </item>
    <item>
     <itemtag>2.</itemtag>
     <itemcontent>A formula<formula> <formulacontent><m:math display="block" xmlns:m="http://www.w3.org/1998/Math/MathML"><m:mrow><m:msup><m:mi>𝛼</m:mi><m:mn>2</m:mn></m:msup><m:mo>+</m:mo><m:msup><m:mi>𝛽</m:mi><m:mn>2</m:mn></m:msup><m:mo>=</m:mo><m:mn>2</m:mn></m:mrow></m:math></formulacontent> </formula></itemcontent>
    </item>
    <item>
     <itemtag>3.</itemtag>
     <itemcontent>A figure<image width="5.000cm" height="3.750cm" id="image-1" name="mill.png" label="3.750cm"></image></itemcontent>
    </item>
   </itemgroup>
   <float detail="figure" chain="figure" implicit="4" explicit="fig:mill" reference="fig:mill">
    <floatcontent><image width="5.000cm" height="3.750cm" id="image-2" name="mill.png" label="3.750cm"></image></floatcontent>
    <floatcaption><floatlabel detail="figure">Figure </floatlabel><floatnumber detail="figure">1</floatnumber> <floattext>This is a Mill</floattext></floatcaption>
    </float>
   <description detail="footnote" chain="footnote" insert="1">
    <descriptiontag><sup>1</sup> </descriptiontag>
    <descriptioncontent>Note that we define a float in <link destination="fig:mill" location="fig:mill">Figure 1</link>.</descriptioncontent>
   </description>
  </sectioncontent>
 </section>
</document>

Note that the math formula has been exported to (formatted for clarity).

<formula> <formulacontent><m:math xmlns:m="http://www.w3.org/1998/Math/MathML" display="block">
  <m:mrow>
    <m:msup><m:mi>𝛼</m:mi><m:mn>2</m:mn></m:msup>
    <m:mo>+</m:mo>
    <m:msup><m:mi>𝛽</m:mi><m:mn>2</m:mn></m:msup>
    <m:mo>=</m:mo>
    <m:mn>2</m:mn>
  </m:mrow>
 </m:math></formulacontent> </formula>

Now, if you want, you can easily filter out the math (or any other part of the document).

Answer 2

I'm a little hesitant to post this as I'm not sure how adaptable it would be to your case, but with the hope that it might point you in the right direction, here it is.

I have been developing a system that allows me to write documents using LaTeX and writes output files as text files in a variety of formats. My current primary use-case is to output HTML5 documents, with the maths as MathML. It's very much in development and I fix the stuff that I need without paying too much attention to the "big picture" of what it might be capable of.

Anyway, a recent adaptation was to use the capabilities of LuaTeX to do the text output directly. Previously I'd been writing to a PDF and using pdftotext to extract the text, but this was unstable. So I played around with LuaTeX a bit and found a way to accomplish what I wanted: direct output to a text file.

The lua file that does this can be found as part of my latex-to-internet repository on github (ignore the README, it is vastly out of date). The specific file is textoutput.lua (there's a fair bit in there which is concerned with converting maths to MathML, I don't know if that's relevant). You'd also need to look at lines 53 to 104 of internettext.code.tex as those install the hooks for outputting to text.

As I said, the system works for me. All of the pages on my website are written as fully-compilable LaTeX documents which my system converts to HTML5+MathML files (the graphics are done with TikZ, naturally). But it would probably need some adaptation to work for your system, particularly as you have no control over the input files.

Answer 3

Partial solution for XML export

In the case of information retrieval, @DavidCarlisle advised me to take a look at LaTeXML. On the website of the project I have found a link to an outdated experiment on exporting ~730k scientific preprints from arXiv.org into XML format. This experiment ended in 2013, but at that moment

Result                  count   %
-------------------------------------
none                    0       0.00
incomplete              40575   5.80
complete with errors    234047  33.46
success                 424800  60.74
-------------------------------------

i.e., only 60% papers pass. Most of the errors are the missing macros and I cannot tell immediately whether this can be fixed.

arXiv is one of my sources of articles, so this is very relevant. Looks like this is not a practical thing, but definitely worth mentioning.