# The Universal Language Processor in TeX: Theoretical Question Regarding the “Expressive” Power of Tex [closed]

How expressive is TeX in terms of being a compiler or a preprocessor of other languages? One splendid example is the `listings` package in which TeX is changed so as to kind of recognize the kind of tokens of almost all programming languages. So, yes, TeX can be made into a kind of preprocessor.

But can TeX be a parser? Can you implement something like YACC in TeX, so that you define a BNF, and then the rest of the input is parsed by this BNF.

Let's simplify the question a bit. is at least theoretically possible (can't cheat by emulating a Turing machine) to make such `lex.tex` file so that the input file

`````` \input lex.tex
DEFINITIONS
letter=[a-zA-Z];
digit=0-9;
alphanum=\$(Letter)|\$(digit);
id=\$(letter)\$(alphanum)*
RULES
id -> { some(\$); }
INPUT
hello world
``````

will actually behave such as a lexical analyzer with three sections: defintions, rules, and input. The first two sections define how the `INPUT` section is processed, producing

``````some(hello) some(world)
``````

If this is possible and realistic in some way, then one may aspire for more, `yacc.tex`, and then perhaps, a universal processor/compiler all written in tex? Wouldn't that be exciting?

• TeX is Turing-complete. While it is possible, it is not immediately realistic. Even if implemented, I imagine such a parser would be incredibly slow. – Sean Allred Feb 5 '17 at 20:05
• I've implemented a more or less complete parser for XML in TeX, including DTD syntax and namespace resolution, for example so can tex be a parser sounds like it ought to have "yes" as an answer, but I didn't implement lex(which is part of a parser generator rather than a parser, doable but tedious) – David Carlisle Feb 5 '17 at 20:07
• while it would be theoretically possible to do in tex, if doing this now it would make far more sense to do it in luatex and use lpeg – David Carlisle Feb 5 '17 at 23:36
• Your question triggered my memory of this answer of mine, tex.stackexchange.com/questions/332012/…. I mention this because the `listofitems` package provides powerful parsing capability, and in the [updated] answer cited, is used to digest alternative syntax for equations and [tries to] convert it to LaTeX, by producing a long string of "spaghetti" or "DNA" code that compiles into the desired result. – Steven B. Segletes Feb 9 '17 at 19:48