1

I have idea of a tool to conveniently skim through arXiv papers/articles. In short, it ought filter out parts of document based on its type (images, tables, formulas, paragraphs etc). Obviously it would be magnitudes of order easier to perform on HTML rather than PDF. AFAIK PDF doesn't contain any info about document structure, basically it's a set of instructions "draw this glyph there", so restoring the structure becomes (an unnecessary) non-trivial OCR problem.

Straightforward solution then is to convert paper .tex source to HTML. Available options are:

  • lwarp: seems like it's for writing with HTML target in mind from outset, requires a lot of source patching
  • latex2html: pretty robust, but some content is converted to images which is not great for responsiveness/a11y
  • pandoc: ditched by arxiv-vanity/engrafo devs; couldn't find any comparisons with other tools
  • make4ht/tex4ht: not sure I understand its internal translation process (.tex -> DVI -> HTML?), overall performs fine with occasional artifacts which I have no idea how to fix (hooking into LuaTeX internals?), but some papers fail to render
  • latexml: there are efforts to process an entire arXiv to HTML, and looking at numbers it's not bad, but haven't tested myself yet; that's what arxiv-vanity uses and from user's perspective conversion quality is on par with tex4ht

Most likely I will use make4ht or latexml after additional testing, but an idea came to me to extract relatively "high level" objects as if I would have thrown out any typesetting; what TeX engine gets after LaTeX content macros are processed. LuaTeX glyph nodes seem to be too low-level for my purpose.

So my question is does LuaTeX/TeX somewhere internally operates on something akin to abstract syntax tree of document?

Improve this question
10
  • 1
    tex doesn't process the whole document so it doesn't really have an ast at all, it is designed for the memory constraints of 1980, the latex macros are only expanded one-by one as the lower level typesetting needs more text, and earlier pages are converted to dvi or pdf and output from memory before later parts of the document are even read from file
    – David Carlisle
    Commented Apr 25, 2021 at 18:18
  • 1
    this is probably relevant latex-project.org/news/2020/11/30/tagged-pdf-FS-study
    – David Carlisle
    Commented Apr 25, 2021 at 18:29
  • 1
    As far as I know LaTeX is concerned with typesetting the text of a document. For this it doesn't need, or have, any concept of an abstract syntax tree, whatever that might be. If you want to "skim" a document then are you considering the initial, or the second, or the third skim? In my GOM opinion I think it best to let the document readers do their own skimming in their own fashion.
    – Peter Wilson
    Commented Apr 25, 2021 at 18:30
  • @PeterWilson in order to make accessible, "tagged" pdf (a legal requirement for some uses) latex will have to generate a "structure tree" annotating the tree structure of the document over the basic typesetting function
    – David Carlisle
    Commented Apr 25, 2021 at 18:33
  • 2
    then this site would be out of a job and we could all go home?
    – David Carlisle
    Commented Apr 25, 2021 at 20:41

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .