Why doesn't TeX have a debugger? And by 'debugger' I mean a dedicated application in which you can set breakpoints and watch the values of registers, macros, etc. as they change in real time, maybe even preview, say, what a box register would look like if it was rendered right now, and try out what an ad-hoc TeX expression would expand to right now, and step through a macro expansion one expansion at a time?
I can see a few ways to ask or interpret the question:
What are the ways that already exist within TeX itself, of debugging macros I write, or typesetting stuff I do (boxes etc.)?
Why isn't there a way to do “proper” debugging: setting breakpoints, watching values change, stepping through program execution (e.g. macro expansion) a step at a time, stepping over (jumping ahead), etc.?
Someone told me that TeX is a programming language; shouldn't it have a debugger like other programming languages do?
For (1), there is a bunch of features in TeX to see diagnostic output:
\tracing macros, packages written to wrap them, etc. See How to best debug LaTeX?, A LaTeX log analyzer application (visualizing TeX expansion), etc. Moving on to (2)…
TeX is a program, and like any program it can be debugged by running it inside a debugger. (For example, programs written in C/C++ can often be run inside debuggers like
gdb and Visual Studio. More on debuggers: list, comparison.)
Donald E. Knuth, the author of TeX, knows the value of debuggers. He wrote the original version of TeX (later called TeX78) in the programming language SAIL, at the Stanford Artificial Intelligence Laboratory, and when programming it he used the debugger available for it. (It was called BAIL. There are mentions of it in
texdoc errorlog.) He loved this debugger so much (IIRC he's raved about it somewhere) that even in the rewritten (current) program, he makes mention of it: if you invoke
texdoc tex you'll see in section 2 (on the very first page after the table of contents) the words:
A complete version of TeX was designed and coded by the author in late 1977 and early 1978; that program, like its prototype, was written in the SAIL language, for which an excellent debugging system was available.
He also knows the value of debugging in general. Even in 1964, when writing the first volume of The Art of Computer Programming, he added these words to the section that begins “We conclude this section by discussing briefly how we might go about writing a complex and lengthy program”:
Step 4 (Debugging). […] Debugging is an art that needs much further study, and the way to approach it is highly dependent on the facilities available at each computer installation. […] The most effective debugging techniques seem to be those that are designed and built into the program itself — many of today’s best programmers will devote nearly half of their programs to facilitating the debugging process in the other half; the first half, which usually consists of fairly straightforward routines that display relevant information in a readable format, will eventually be thrown away, but the net result is a surprising gain in productivity.
Accordingly, in the TeX program, in addition to the various
\tracing routines (see (1) above), he added, in appropriate places, various special routines to the TeX program (as he did to many of his programs) that are meant to be invoked from the debugger. These are marked inside debug…gubed sections in the TeX program. (This is a macro that is defined as "nothing" in debug mode, and as begin-comment / end-comment in non-debug mode.) If you look in the index to the TeX program under the entry for
debug, you'll see that it is used in sections 7, 9, 78, 84, 93, 114, 165, 166, 167, 172, 1031, 1338 of the TeX program. The last of these starts Part 52 of the TeX program, which is titled Debugging:
Debugging. Once TeX is working, you should be able to diagnose most errors with the
\showcommands and other diagnostic features. But for the initial stages of debugging, and for the revelation of really deep mysteries, you can compile TeX with a few more aids, including the Pascal runtime checks and its debugger. An additional routine called debug_help will also come into play when you type ‘
D’ after an error message; debug_help also occurs just before a fatal error causes TeX to succumb.
The interface to debug_help is primitive, but it is good enough when used with a Pascal debugger that allows you to set breakpoints and to read variables and change their values. After getting the prompt ‘
debug #’, you type either a negative number (this exits debug_help), or zero (this goes to a location where you can set a breakpoint, thereby entering into dialog with the Pascal debugger), or a positive number m followed by an argument n […]
So if TeX were built in debug mode, then at the prompt where TeX shows
Type <return> to proceed, S to scroll future error messages, R to run without stopping, Q to run quietly, I to insert something, 1 or ... or 9 to ignore the next 1 to 9 tokens of input, H for help, X to quit.
if you hit the secret option
D, it would enter the debugger where you could use all the regular functionality of your debugger plus the ones Knuth added. (See section 84 of the TeX program.) Don't bother trying it in your TeX distributed by TeX Live / MikTeX (etc.) though: unfortunately, since the late 80s or so, TeX isn't run as a Pascal program that can be run with a Pascal debugger. It has been mainly built via
web2c (originally written by Tomas Rokicki), a translation into C that loses much of these debugging aids.
Still, it is possible to regain some debugging, if you build TeX yourself. See this post “Looking inside TeX: C helps me to see” by Graham Douglas (who recently joined Overleaf and has written some great posts on their blog): in this post from 2014 he describes building TeX in such a way to run it inside a debugger. (See also this answer by him on this site.) He writes:
Although I have quite a collection of books on TeX, I've always found it really, really hard to understand how TeX – the language and program – actually works. So, for me, I find it much more instructive to watch how some bits of TeX actually work by stepping through the C code as TeX is executing – single-stepping via the Visual Studio interface. […]
Despite the difficulty in following the execution of TeX.C, it is nevertheless fascinating to watch TeX actually run: Parsing the input file, acting on catcode values, creating tokens, defining macros, building boxes, running the page-builder and shipping out pages. Although I'm only just starting to explore TeX via C code, it has, for me, started to lift some of the confusion surrounding the TeX language – even if I have barely scratched the surface of this truly extraordinary program.
This I think is exactly what you asked for.
Personally I think it's a tragedy that this sort of “seeing” TeX is not accessible to more users. (I have some ideas for improving this that once I thought I'd work on, but realistically speaking it's a lot of work and I'll probably never get around to it… a wild hope that someone does!)
For (3): no, TeX is not a programming language, whatever anyone may tell you. It is a program for typesetting, which includes a macro expansion system, and this macro system happens to be Turing-complete and possible to program in, but that doesn't necessarily mean it's a good idea. Yes Lamport came up with some TeX programming hacks and built an elaborate edifice using them to realize his vision of a document preparation system, and others have built on it further. But such programming wasn't Knuth's idea and even when he wrote non-trivial macros he did it with a certain amount of reluctance:
TeX was designed for typesetting, not for programming; so it is at best “weird” when considered as a programming language. — DEK, Digital Typography, page 235
It happens sometimes that some features added to a program end up becoming themselves a programming language (see Accidentally Turing-Complete, surprisingly Turing-complete): it's possible to write programs in Apache's
mod_rewrite rules, in the Haskell type system, and so on. Programming in TeX macros is, well… not so out-of-the-way, but still requires a certain degree of perversity. :-)
The way TeX macro expansion works can be hard to understand and to explain, as Knuth found out when he taught a course on the TeX program. (The TeX program source code was the textbook for the course.) As a result, he added some more diagnostic output and called it a new program:
The answer to this problem was much more difficult to explain in class than I had thought it would be, so I guess it was also much more difficult for the students to solve than I had thought it would be. After my first attempt to explain the answer, I decided to make up a special version of TeX that would help to clarify the scanning routines. This special program, called DemoTeX, is just like ordinary TeX except that if
\tracingstats>2the user is able to watch TeX's syntax routines in slow motion.
The format of this output is a bit cryptic as it has to show everything on the terminal, but if you free yourself from that restriction you can make it more readable. I've played with something like this locally and it can be quite informative. Again, unfortunately, not something available easily, but possible in principle.