8

I've been programming quite a bit in expl3 for the last weeks, and while I really like the expressiveness of the provided API, what often is incredibly painful is debugging larger programs.

In many circumstances, I find the various \xxx_show: functions very useful to query the values of variables and see what's going on at execution. For more readable debugging output \iow_term:x and other I/O functions can be used. l3basics also provides some built-in debugging facilities to check for assignments to undeclared variables, wrong arithmetic expressions and a few other things, which I never felt pretty useful, though.

Sometimes all those do not help anymore when things like expansions go wrong and TeX aborts with an utterly unhelpful low-level error message. In TeX or LaTeX2e using \tracingmacros or \tracingcommands often reveals the problem quickly.

In expl3, however, the tracing output is almost always too verbose to be actually helpful. For example, if I enable macro tracing in the example document from my question about parsing balanced groups only outside of all cctab functions, I get about 7600 lines of log output. If the few cctab function calls are included into the tracing, the log file blows up to 120k lines! Similar things happen for seemingly innocent functions like the \xxx_show: functions. It's no fun to scroll around in hundreds and hundreds of lines of code just to locate one of the functions you actually called in your program.

So are there other tools to aid in debugging of, in particular, expl3 programs/package code? Or is there a way to selectively mute the tracing output for specific sets of functions, e.g. all kernel functions? If not, what plans are there to support more debugging facilities in future expl3 versions?

3
  • 3
    Oftentimes I find \unravel from the unravel package very useful, but it's not perfect and doesn't support every aspect of TeX. Breaking the code in smaller parts and checking whether each part does exactly what you think it does is generally a good habit. And the naive debugging variant of putting \typeout (or similar) all over the place to see at which point the error happens :) Else rubber-ducking is always a useful approach.
    – Skillmon
    Nov 16, 2019 at 8:47
  • 1
    "There are three things a man must do before his life is done; write two lines in expl3 and make the buggers run." (paraphrased from the Devil's DP Dictionary) Nov 16, 2019 at 14:55
  • @JohnKormylo Do ExplSyntaxOn and ExplSyntaxOff count? ;-) Nov 16, 2019 at 15:29

1 Answer 1

7

Too long for a comment, and too vague for an answer; feel free to downvote ;-)

Unfortunately, no. The options you listed are more or less all there's available. One option, which you didn't mention, is to use \traceon/\traceoff from the trace package, which allows you to selectively turn off tracing for uninteresting bits of code (l3msg functions are notoriously log-y, so they are wrapped in \conditionally@traceoff and \conditionally@traceon to work well with trace). So if you are debugging some code and you know for a fact that some part of it is working (say, starting a catcode table), you can do something like this:

% with \usepackage{trace}
\traceon
% some code to be debugged
\conditionally@traceoff
% log-exhaustive code which you are sure it's working
\conditionally@traceon
% more code to be debugged
\traceoff

but then, there's no much improvement on the overall because few functions actually support trace (that I remember: NFSS, calc, xparse, and l3msg).

The kernel could have more occurrences of \conditionally@traceoff and \conditionally@traceon here and there to improve the situation, but this comes at the price of some performance. Another major drawback is that with functions which work with expansion only, you simply can't start or stop tracing.


This deficiency of a good debugging tool is unlikely to be solved (at the l3kernel level or else) because the underlying TeX engine provides either no debug or far too many debug, and the information it shows is only (easily) available at the engine level.

Also, slightly different from "usual" languages, what you are debugging impacts on what the debugging tool should look like. If you are debugging some text which is passed around in non-expandable code, usually \showtokens or something like that is really helpful, if you know where to look for the problem. If not, then you're back at \tracingall.


When I am trying to debug some code I usually try to isolate the problem to the smallest possible part of the code to get the smallest possible logging. In this process I usually end up removing all code that is not mine (usually I am to blame for the problem), so this shortens the log by a lot. Even within my own parts of the code, I try to pinpoint the problem to a single macro call when possible.

When I can't narrow the problem down anymore I usually use \showtokens on the arguments of a macro or \show in control sequences to inspect their contents. If I'm in an expansion-only context, then I use a trick similar to \msg_expandable_error:nn to make an expandable \showtokens:

\def\eshow#1{\expandafter\@gobble\expandafter{\ERROR <#1>}}

If none of this works, then I go for \traceon plus some regex searching through the log (don't try to debug looking at the logs in the terminal: you'll find yourself hopelessly scrolling up and down and forgetting what you were searching to begin with :-).

Finally, if the problem depends on expansion tricks (mainly if you rely on manipulating the input token stream) then \traceon will also be useless, most of the time, because it doesn't show the token stream at all. In those cases, when all hope is lost, I resort to unravel (I'm not implying it's bad, on the contrary, it's too good, and small pieces of code take thousands of steps to complete). \unravel will usually show you clearly where something went wrong, but it will test your patience first ;-)


If nothing else works, stop writing macros and just write some paragraphs of text for a change ;-)

6
  • I had actually used trace in the past (forgot to mention it), but I sometimes also found a bit too verbose, so I switched back to the native \tracing methods. expl3 aleady has an enable-debug option which allows for extra sanity checks. Wouldn't it be possible to have another option, enable-trace or something, which adds several tracing category switches when set? You could then selectively enabled/disable kernel tracing, font handling, string wrapping and such. I don't know the internals of LaTeX3 well enough to see if this would work in theory.
    – siracusa
    Nov 16, 2019 at 13:33
  • @siracusa The enable-debug option, if I can do a crappy analogy to a compiled language, is more of a compile-time debug (mostly to catch if you're following variable naming conventions, i.e. \l_..., \g_..., and if you're not using deprecated functions), at the expense of a significant performance hit. What you want is more of a runtime checking, which is much more complicated. In theory, yes, something like \conditionally@traceoff could be added to the kernel's (non-expandable, at least) functions, but this would need to be a patch rather than a fixed feature due to performance... Nov 16, 2019 at 13:41
  • What do you mean by "a patch rather than a fixed feature"? I was thinking of something like what \__kernel_if_debug:TF does, conditionally define different functions when the package is loaded. This would clearly bloat the code, though, as for many functions (at least) two definitions had to be provided.
    – siracusa
    Nov 16, 2019 at 14:38
  • @siracusa What I meant is that the tracing code would have to be a package-loading option, which would change the definition of the kernel functions, rather than something that is always checked when a function is executed. I said patch because currently the debug code is not inserted as a "define like this or like that", but as a patch (take a look here and the lines that follow). Personally would be in favour of an l3trace module to make debugging easier, but such module would [...] Nov 16, 2019 at 15:24
  • @siracusa [...] have to be a l3-non-trace module instead, because you usually want to debug your code, not expl3's. Probably such module would just insert a bunch of \conditionally@traceoff here and there so that tracing kernel functions would be usually off so that you can focus on your code instead... Nov 16, 2019 at 15:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .