# Accessing TeX token or exporting filtered plain text

(updated according to gained information)

In the context of keyword extraction from arbitrary (La)TeX documents and simple manipulations with them (automated annotation), I would like to be able to do the following things:

1. Access a stream of tokens with information required to build tree-like representation (i.e., I need to know when environment starts, when it ends, when math formula occurs)
2. Export plain text representation of the document while being able to filter out commands (for example, remove equations or some sections).

I already have a very complicated LaTeX parser written in C++, but still I am unable to avoid the problem of expansion of commands.

Basically, if an author defines a command \keyword that just prints keyword, I cannot deal with this in a general manner. (And people really do that and tend to do this with the most important keywords!)

So, on one hand I need to get a plain text representation of the document while filtering out the math statements and not loosing information about position of the commands and environments (to aid sentence tokenizer and keyword extractor).

On the other hand, having an arbitrary TeX document I need to find a node with a specific word in it, manipulate it (remove/replace/wrap in a link) and then compile the document.

I vaguely understand that this should be possible using LuaTeX or ConTeXt, but I wasn't able to find accessible examples on LuaTeX wiki.

Update: I think I need to clarify that I cannot control the incoming documents, so the goal is to analyse and manipulate arbitrary valid LaTeX code.

Bounty update: Existing answer provides lots of valuable information about achieving my goals with writing input documents in ConTeXt, which is unfortunately inapplicable

Bounty update 2: To provide you the scale: we need to process more than a million documents and the number is increasing.

• you would have to define some tree structure, TeX doesn't build anything approximating an AST representation that you could access – David Carlisle Jun 1 '16 at 16:16
• @DavidCarlisle true, I understand that TeX does not function in this way. But by a stream of different tokens and state switches I can build such a tree. Anyhow, AST is how I do things now, but this is not the only way – Andrii Magalich Jun 1 '16 at 19:10
• there are no compound tokens like words in tex, there are just sequences of character tokens at the token level or sequences of character/glyph nodes in a horizontal list in a box. Of the lua callbacks you mention, the node one is necessarily after expansion. For doing weird replacement effects to text in lualatex look at the chickenize package which has examples of the (ab)use of many of the callbacks into luatex's node structure. – David Carlisle Jun 3 '16 at 0:44
• @AndriiMagalich no really it never knows that, it does not parse the document in advance it interleaves tokenisation, and execution so has usually only read as far as the next line, so is executing the content of an environment before it has seen the end. – David Carlisle Jun 15 '16 at 15:33
• @AndriiMagalich that is what the existing convertors such as tex4ht or latexml (probably try that first) do, but for example if someone goes \def\keyword{hello}\keyword I would expect all such convertors to make that indistinguishable from hello isn't that what you want? – David Carlisle Jun 15 '16 at 15:39

In ConTeXt, one option is to export your document as XML. You can then manipulate the resulting XML document using standard XML tools in your favourite programming language.

As an example, consider this example:

\setupbackend[export=yes]
\setupinteraction[state=start]

\starttext
\startsection[title={Test of export}]
This is to test how \CONTEXT\ export works. First, we input a test file
below.

\startnarrower
\input ward
\stopnarrower

Let's see some other environments, such as\footnote{Note that we define a
float in \in{Figure}[fig:mill].}

\startitemize[n]
\item A table:

\startTABLE
\NC 1 \NC 2 \NC 3 \NC \NR
\NC 1 \NC 2 \NC 3 \NC \NR
\NC 1 \NC 2 \NC 3 \NC \NR
\stopTABLE

\item A formula

\startformula
α^2 + β^2 = 2
\stopformula

\item A figure

\externalfigure[mill.png][width=5cm]

\stopitemize

\startplacefigure
[title={This is a Mill}, reference={fig:mill}]
\externalfigure[mill.png][width=5cm]
\stopplacefigure

\stopsection
\stoptext


and process it using context filename. This will create a directory \jobname-export with the following structure:

\jobname-export
|
+-- images
|   +--- <image file1>
|   +---- ....
+-- style
|   +-- \jobname-defaults.css
|   +-- \jobname-images.css
|   +-- \jobname-styles.css
|   +-- \jobname-template.css
+-- \jobname-div.xhtml
+-- \jobname-pub.lua
+-- \jobname-raw.xml
+-- \jobname-tag.xhtml


Out of these, the \jobname-raw.xml contains the AST of the complete document. For the above document it is:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>

<!--

input filename   : test
processing date  : Tue Jun 14 19:58:26 2016
context version  : 2016.06.02 21:28
exporter version : 0.34

-->

<?xml-stylesheet type="text/css" href="styles/test-defaults.css" ?>
<?xml-stylesheet type="text/css" href="styles/test-images.css" ?>
<?xml-stylesheet type="text/css" href="styles/test-styles.css" ?>

<document date="Tue Jun 14 19:58:26 2016" file="test" context="2016.06.02 21:28" language="en" xmlns:m="http://www.w3.org/1998/Math/MathML" version="0.34">
<section detail="section" chain="section" implicit="1" level="3">
<sectionnumber>1</sectionnumber>
<sectiontitle>Test of export</sectiontitle>
<sectioncontent>
This is to test how ConTEXt export works. First, we input a test file below.
<break/>
The Earth, as a habitat for animal life, is in old age and has a fatal illness. Several, in fact. It would be happening whether humans had ever evolved or not. But our presence is like the effect of an old-age patient who smokes many packs of cigarettes per day <subsentencesymbol symbol="middle">—</subsentencesymbol> and we humans are the cigarettes.
<break/>
Let’s see some other environments, such as<descriptionsymbol detail="footnote" insert="1"><sup>1</sup></descriptionsymbol>
<itemgroup detail="itemize" chain="itemize" level="1" symbol="n">
<item>
<itemtag>1.</itemtag>
<itemcontent>A table:<table><tablerow><tablecell align="flushleft"> 1 </tablecell><tablecell align="flushleft"> 2 </tablecell><tablecell align="flushleft"> 3 </tablecell></tablerow><tablerow><tablecell align="flushleft"> 1 </tablecell><tablecell align="flushleft"> 2 </tablecell><tablecell align="flushleft"> 3 </tablecell></tablerow><tablerow><tablecell align="flushleft"> 1 </tablecell><tablecell align="flushleft"> 2 </tablecell><tablecell align="flushleft"> 3 </tablecell></tablerow></table></itemcontent>
</item>
<item>
<itemtag>2.</itemtag>
<itemcontent>A formula<formula> <formulacontent><m:math display="block" xmlns:m="http://www.w3.org/1998/Math/MathML"><m:mrow><m:msup><m:mi>𝛼</m:mi><m:mn>2</m:mn></m:msup><m:mo>+</m:mo><m:msup><m:mi>𝛽</m:mi><m:mn>2</m:mn></m:msup><m:mo>=</m:mo><m:mn>2</m:mn></m:mrow></m:math></formulacontent> </formula></itemcontent>
</item>
<item>
<itemtag>3.</itemtag>
<itemcontent>A figure<image width="5.000cm" height="3.750cm" id="image-1" name="mill.png" label="3.750cm"></image></itemcontent>
</item>
</itemgroup>
<float detail="figure" chain="figure" implicit="4" explicit="fig:mill" reference="fig:mill">
<floatcontent><image width="5.000cm" height="3.750cm" id="image-2" name="mill.png" label="3.750cm"></image></floatcontent>
<floatcaption><floatlabel detail="figure">Figure </floatlabel><floatnumber detail="figure">1</floatnumber> <floattext>This is a Mill</floattext></floatcaption>
</float>
<description detail="footnote" chain="footnote" insert="1">
<descriptiontag><sup>1</sup> </descriptiontag>
</description>
</sectioncontent>
</section>
</document>


Note that the math formula has been exported to (formatted for clarity).

<formula> <formulacontent><m:math xmlns:m="http://www.w3.org/1998/Math/MathML" display="block">
<m:mrow>
<m:msup><m:mi>𝛼</m:mi><m:mn>2</m:mn></m:msup>
<m:mo>+</m:mo>
<m:msup><m:mi>𝛽</m:mi><m:mn>2</m:mn></m:msup>
<m:mo>=</m:mo>
<m:mn>2</m:mn>
</m:mrow>
</m:math></formulacontent> </formula>


Now, if you want, you can easily filter out the math (or any other part of the document).

• Thank you, this is excellent! Do you have any knowledge regarding the reliability of this conversion? I.e., will it work with an arbitrary LaTeX document using arbitrary packages (but, of course, compilable in the environment) – Andrii Magalich Jun 2 '16 at 12:35
• This is a ConTeXt solution, so it won't work with LaTeX packages. It is relatively simple to make a new environment compatible with this export. – Aditya Jun 2 '16 at 20:56
• I see... Unfortunately, input documents are given and the whole thing is that I cannot control them. – Andrii Magalich Jun 2 '16 at 21:34
• Please don't delete. – morbusg Jun 3 '16 at 16:01
• Please retain the answer: it demonstrates it's technically doable! Most of the issue with translating to LaTeX (or plain or ...) is that ConTeXt has structures at the macro level to allow this to work: the Lua side is not the 'key' to the problem. – Joseph Wright Jun 14 '16 at 20:15

I'm a little hesitant to post this as I'm not sure how adaptable it would be to your case, but with the hope that it might point you in the right direction, here it is.

I have been developing a system that allows me to write documents using LaTeX and writes output files as text files in a variety of formats. My current primary use-case is to output HTML5 documents, with the maths as MathML. It's very much in development and I fix the stuff that I need without paying too much attention to the "big picture" of what it might be capable of.

Anyway, a recent adaptation was to use the capabilities of LuaTeX to do the text output directly. Previously I'd been writing to a PDF and using pdftotext to extract the text, but this was unstable. So I played around with LuaTeX a bit and found a way to accomplish what I wanted: direct output to a text file.

The lua file that does this can be found as part of my latex-to-internet repository on github (ignore the README, it is vastly out of date). The specific file is textoutput.lua (there's a fair bit in there which is concerned with converting maths to MathML, I don't know if that's relevant). You'd also need to look at lines 53 to 104 of internettext.code.tex as those install the hooks for outputting to text.

As I said, the system works for me. All of the pages on my website are written as fully-compilable LaTeX documents which my system converts to HTML5+MathML files (the graphics are done with TikZ, naturally). But it would probably need some adaptation to work for your system, particularly as you have no control over the input files.

• So, you inject a small snippet of code in the preambule of the document and it allows you to parse the structure of the page? Did you try this with complicated documents like scientific papers? – Andrii Magalich Jun 15 '16 at 7:21
• @AndriiMagalich It's a bit more complicated than that. The system was written for me to use and I don't really care about mass translating documents so I didn't write it for that. I just wanted to show an example of getting TeX to do some processing and get text output. – Loop Space Jun 15 '16 at 18:50
• I am just curious, why not use tex4ht? Since you say My current primary use-case is to output HTML5 documents, with the maths as MathML, could not tex4ht be used for this? Just asking, that is all. Thanks. – Nasser Jun 30 '16 at 4:55
• @Nasser That's a good question, comment prob a bit short for a full answer. First, html wasn't my initial goal. I was originally using it to author articles for other formats such as Markdown and its many derivatives. I wrote blog posts for the TeX blog using it, for example. Second, I found I wanted more control over the detail than tex4ht could easily give me. Rather than trying to get a webpage to look like a pdf, I wanted to be able to easily control different aspects of the webpage by configuring the document accordingly. tex4ht seemed too much like a black box for what I wanted. – Loop Space Jun 30 '16 at 6:22

## Partial solution for XML export

In the case of information retrieval, @DavidCarlisle advised me to take a look at LaTeXML. On the website of the project I have found a link to an outdated experiment on exporting ~730k scientific preprints from arXiv.org into XML format. This experiment ended in 2013, but at that moment

Result                  count   %
-------------------------------------
none                    0       0.00
incomplete              40575   5.80
complete with errors    234047  33.46
success                 424800  60.74
-------------------------------------


i.e., only 60% papers pass. Most of the errors are the missing macros and I cannot tell immediately whether this can be fixed.

arXiv is one of my sources of articles, so this is very relevant. Looks like this is not a practical thing, but definitely worth mentioning.

• you may also look at tex4ht, which supports most LaTeX packages and it is highly configurable, so it is easy to add support for missing commands and environments – michal.h21 Jun 15 '16 at 17:45
• @michal.h21 thanks, David mentioned it as well, but I haven't looked at it yet. I looks really rusty – Andrii Magalich Jun 15 '16 at 17:56
• it is not user friendly at first sight, but it is powerful and it has extensive infrastructure for custom configuration – michal.h21 Jun 15 '16 at 18:04
• 60% strikes me as very good, considering. However, note that arXiv does not accept just any valid LaTeX source. So there's going to be a partial standardisation effect as people have to made their code compliant in order to get it accepted.Neither can teX4ht manage all valid LaTeX, although it does extremely well. It may look 'really rusty' to you - it is not currently maintained - but it is better than any of the other converters I've tried. – cfr Jun 16 '16 at 1:37