TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I recently asked a related question about how often one should update LaTeX packages. Supposing that I update a bunch of packages in one go, or regularly update with only a few packages changing, how can I test that my TeX documents aren't changing substantially?

In programming, this would be akin to unit or regression testing. Maybe I am bringing too much experiences with programming bugs to the LaTeX realm, but it is a concern of mine that, over time, different versions of packages will lead to different PDF output.

I am aware of latexdiff, and use it routinely for manual inspection of differences in tex files, but that doesn't address two issues: it won't easily capture differences attributable to different package versions, and it isn't clear to me that it's easily automated to identify those files that have differences. It seems to me that the former issue requires that I do a lot to manually segregate different versions of packages, and the latter issue requires doesn't seem easily automated at all.

Is there another method for checking that the output is visually the same, for files compiled with different versions of the same dependencies?

Update 1: A use case: Suppose you have a lot of published documents or a large manuscript, and want to be sure that their compiled versions haven't changed due to a package update. It may be the case that you want to change the output, hence the reason for a package update, but it's most important that you know if anything has changed, and you can subsequently discern how and why they changed.

share|improve this question
You also have to allow for differences that are deliberate ... – Joseph Wright Feb 28 '12 at 20:00
@JosephWright I'm not entirely clear what you mean. I'd like to have all differences reported to me... I'll include a use case to help. – Iterator Feb 28 '12 at 20:04
My point was that if a package has been updated then unless it was only a documentation fix presumably something will be different. – Joseph Wright Feb 28 '12 at 20:10

The LaTeX3 Project has some experience in this area, although most of our automated tests are currently focussed more on code input/output than typeset material. These scripts were developed for LaTeX2e, but have been somewhat updated for work on LaTeX3.

In the context of working with boxes and the output routine, we do have two forms of regression testing that do test typeset material, at least to some extent. Both these and the more programmatic tests rely on using one or more test documents which write verbose information to the log file. After some script-based cleanup to remove 'dynamic' material, these can then be compared with a reference version of the same log, to check the input and output are the same.

To construct appropriate test files, some imagination may be needed. For example, for box-based test we use something akin to \showthe\box..., which will show the content of a box in the log. Thus if you box up the material to be checked, then write the box to the log, you can see exactly what is there.

For full pages, you can use \tracingall during shipout to record everything on a page in the log. Complementing this, we also have 'trace' DocStrip guards in, for example, the output routine code. This extra code writes very detailed information to the log, for example every outcome of a calculation. Change to the layout will alter these numbers, if only subtly, and this can then be picked up by a script.

The implementation that the LaTeX3 Project use for testing is all available from our GitHub site. The key mechanisms are a special LaTeX file which is used by the typesetting tests, and a small Perl script which formats the logs (at some stage, it would be nice to rewrite the latter in Lua, as it would keep things more self-contained). Supporting these two files are a number of Makefile/batch scripts, all of the same general form, and the various test files themselves (test input as .lvt, test output as .tlg).

share|improve this answer
Frank Mittelbach is very keen to get more tests written for LaTeX3. The problem is that it's not easy, and there are lots of jobs to do! – Joseph Wright Feb 28 '12 at 20:11

This is not really a complete answer, just some thoughts and hints. On the other hand, it is too long for a comment.

So given that we have updated a couple of packages:

  • A first regression could be to compile all test documents and see if there is any diff in the compilation process itself by comparing the logfiles. New compilation errors do obviously indicate a regression problem, but new warnings might also be helpful. Of course, the logged output by some package might have changed without any "substantial change" to the package itself. However, grep`ing for "magic terms", such as "depricated" might nevertheless trigger potential regression issues.
  • Given that the documents still compile, the important point is, of course, the visual difference. An obvious approach is to compare the resulting PDFs, page by page. Unfortunately, very few "PDF diff tools" actually compute a visual diff. AFAIK, Adobe Acrobat Pro can do this. A bit more scripting-friendly (even though Acrobat is said to be well scriptable) might be a solution that involves ImageMagick or some other converter to first convert each PDF page-wise into PNG graphics. The PNG pages (in theory) should then be binary "diffable". At least one should be able to find pages with some visual difference.
  • Depending on your set of "test documents", the previous point might involve a high computational overhead. If you have a couple of books that, all together, make up some thousand pages, a blind page-wise comparison of everything will be expensive. It might be a good idea to use the -recorder option of pdflatex to compute for each document a list of the actually used input files (including packages and all their dependencies) and to use this information to do the regression only with those documents that actually depend on a changed package.
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.