# Can LaTeX be persuaded to produce text output?

I've read here and there that ConTeXt can produce XML output. We also have, from time to time, questions about converting LaTeX to different formats. On the basis that "The only parser for TeX is tex", if latex could produce text output instead of PDF then it would be possible to write a style file to convert reasonable input to a different markup language.

Would this be possible?

Bit of background: I encounter this "can we convert from LaTeX?" question in the context of the nLab where the input format is Markdown+iTeX (iTeX not being anything to do with Knuth's proposal but a subset-of-LaTeX-to-MathML converter) but people often have snippets of LaTeX articles that they want to include. So converting all the way to XHTML+MathML via, say, tex4ht isn't the right option. I wrote a Perl script that reimplements much of TeX to do this, but after doing so realised that my style files would work in ordinary LaTeX and produce the "right" output, except that they would be embedded in a PDF. So if I could just persuade TeX to produce text, I'd be almost there. Of course, I could try to extract the text from the PDF but that "feels wrong" and I'd worry about extra stuff sneaking in by accident.

-
What's wrong with a simple pdftotext postprocessor. Extra stuff can sneak in by accident in whatever solution you try. – Aditya May 31 '11 at 18:59
@Aditya: I know, but I feel that I have more understanding of what TeX can produce itself. If it is a genuine text file, then I'd know what to filter out afterwards. It's partially to do with my own lack of understanding of things like the PDF format. I guess what I want is for TeX to send all "printable" characters to a "write". – Loop Space May 31 '11 at 20:23
@Aditya: How does ConTeXt do this? (I got the quip about parsing TeX from your blog, by the way.) – Loop Space May 31 '11 at 20:23
I am not 100% sure how ConTeXt does it. The code is in back-exp.lua. IIUC, it builds a tree of the entire document in memory...each macro and environment defined using \define and \definestartstops hooks into that tree; all the core envuronments (itemize, enumerate, section, etc) hook into the tree. Then, at the end of the document, ConTeXt simply serializes the tree and writes it to a separate text file. – Aditya May 31 '11 at 21:34
@Aditya: I've now implemented something that uses pdftotext as the last stage ... and it works! I was able to produce the source text to this page: ncatlab.org/nlab/show/equivariant+tubular+neighbourhoods by writing a LaTeX document with a special style file. So could you assemble your various comments in to an answer which I can accept? (If it's alright by you, after you've done that then I might add some details on exactly what I did, but I'd like to give you the credit for the solution.) – Loop Space Jun 21 '11 at 21:45

The underlying solution is of course the same for ConTeXt and LaTeX: you need to have a way of changing what macros do such that they write the correct output rather than typesetting. This is also much the same as tex4ht does. The advantage ConTeXt has is that the macros are provided mainly by one focussed group, and they include the necessary 'back end' to make that conversion easy. To do the same for LaTeX, you need to handle all of the macros that might be present, which is a problem given the number and variety of LaTeX packages. So while in principal it's possible, the implementation is a severe challenge.

(With my 'LaTeX3 hat' on, this is an obvious area to bear in mind when defining an updated format. To do that, you need to have a much more 'regular' syntax and input than is often the case with LaTeX files at present. Again, I think ConTeXt shows how this can be done as it is already good on keeping the input within it's own structures.)

-
I guess the key point is it partly depends on what package coverage you want. Doing the LaTeX2e kernel, with no hacks in the source, would be relatively straight-forward. Throw in some low-level TeX, a few kernel hacks and a load of packages, and things are different. – Joseph Wright May 31 '11 at 17:31
Changing the macros is fine. I've basically already done that as part of my Perl script, and I can add more on a case-by-case basis (I'm not looking for a global solution). It's the fundamental getting LaTeX to produce ASCII text rather than DVI or PDF or ... that I'm interested in here. As I understand it, tex4ht works by adding hooks in to the DVI or PDF and post-processing it. I'd prefer to simply have tex output the text directly. – Loop Space May 31 '11 at 17:48
@Andrew: Ah, I see. – Joseph Wright May 31 '11 at 17:59
A blasphemous suggestion: write a few TeX (rather ConTeXt) macros that can read LaTeX syntax and convert them to ConTeXt Apart from syntax for references, other things is LaTeX are easily convertible to ConTeXt syntax. There was an old project with Brookes Moses (I think) that tried to do this. If you can find that, then most of the ground work is already done. – Aditya May 31 '11 at 21:40
@Aditya: Yes, that was my project. It wasn't that hard to do, just a bit tedious, and I didn't really get all that far with it -- but what I have is here: files.dpdx.net/tex/context/latex-compat. I hadn't thought about it in years; I'm amazed that someone remembers it. – Brooks Moses Jun 2 '11 at 15:59

It is possible to achieve what you want, provided you do not want TeX to act as a parser. In my opinion, part of the success of TeX, is that it has managed to transform itself over the years to act as a language transformation tool. First it was TeX->Postscript and now it is TeX->pdf. Tralics has been fairly successful to produce TeX->XML.

But, I think one needs to look at the problem from a different angle. With todays available technologies one, needs to have a "Universal Mark-up Language". Markdown and Yaml are scaled down tools and can never be able to be full document description languages, so going that route will limit one's efforts.

Sometime back, I designed a CMS based on text files. All mark-up was in plain text and fragments from Wikipedia's markup language. I would load the text file via php and then filter the input and produce the HTML page.

<!--
{{feature-image: http://localhost/images/sample102.jpg }}
{{feature: A collection is like a puzzle...}}
-->


The feature-image was a div and the feature-text the caption. I had commands for image-credits and the like.

Now this is not so difficult to produce with TeX. So my proposal is to actually use TeX to write an intermediate mark-up in a text file then parse with your language of choice to achieve what you wish.

Workflow depending on targets can be one of the following:

   TeX->Intermediate MarkUp->HTML
TeX->pdf
TeX->plain text
Intermediate MarkUp->Translator (javascript, perl, python,


In a nutshell, retain TeX and output into a new mark-up language. Markdown and other technologies can be a subset of this.

\documentclass{article}
\usepackage[demo]{graphicx}
\usepackage{verbdef}
\begin{document}
\makeatletter
%% create file and open it to write
\newwrite\file
\immediate\openout\file=wikimark.wiki
\newif\if@wikimark
\newif\if@html
\@wikimarktrue

\def\image#1#2{%
\if@wikimark
\image@@{#1}{#2}
\else
\includegraphics{dummy.png}
\fi
}

\def\Section#1{%
\if@wikimark
\section@@{#1}\relax
\else
\section{#1}
\fi
}

\def\image@@#1#2{%
\immediate\write\file{\string{\string{img:#1\string}\string}}
\immediate\write\file{\string{\string{img-caption:#2\string}\string}}
}

\edef\hash@@{\string#\string#}

\def\section@@#1{%
\immediate\write\file{\hash@@ #1}
}

\makeatother

\Section{Test Section}

\image{http://tex.stackexchange.com/questions/15440/parsing-files-through-lua-tex}{This is the caption}

\closeout\file
\end{document}


The minimal is just a proof of concept. Main idea here is not to redefine the LaTeX commands but rather add new ones with switches for other mark-up.

-
That sounds pretty similar to what I want, apart from that I don't understand your statement "provided you do not want TeX to act as a parser". I want TeX to properly expand its input, acting accordingly, but I want the output file format to be text. I can do this with my perl version of TeX, but after writing it I felt a better solution would be possible if only I could make TeX output actual text. – Loop Space May 31 '11 at 18:46
@Andrew Stacey What I mean as long as one does not expect TeX to read a file with alien mark-up like html and try to convert. Everytime TeX writes to an auxiliary file or a log it writes text! PGF can export a table to html. To transform to mathml I would add the tags, and include the maths verbatim and write to the file straight. Going the LuaTeX way might make things easier with the file operations. – Yiannis Lazarides May 31 '11 at 18:57
Yiannis: Right, that's fine. All I want is for TeX to process TeX, nothing more. I feel that I can handle the actual conversion bit. It's the sheer mechanics of outputting the main stream to a text file that I don't know how to do. – Loop Space May 31 '11 at 20:24
@Andrew OK, I will post a minimal in the morning, getting too late here! – Yiannis Lazarides May 31 '11 at 21:05
Yiannis: It looks promising, but how does the ordinary text get in to the new file? – Loop Space Jun 1 '11 at 9:05