How can I automate “regression testing” for LaTeX documents, when updating packages?

Question

I recently asked a related question about how often one should update LaTeX packages. Supposing that I update a bunch of packages in one go, or regularly update with only a few packages changing, how can I test that my TeX documents aren't changing substantially?

In programming, this would be akin to unit or regression testing. Maybe I am bringing too much experiences with programming bugs to the LaTeX realm, but it is a concern of mine that, over time, different versions of packages will lead to different PDF output.

I am aware of latexdiff, and use it routinely for manual inspection of differences in tex files, but that doesn't address two issues: it won't easily capture differences attributable to different package versions, and it isn't clear to me that it's easily automated to identify those files that have differences. It seems to me that the former issue requires that I do a lot to manually segregate different versions of packages, and the latter issue requires doesn't seem easily automated at all.

Is there another method for checking that the output is visually the same, for files compiled with different versions of the same dependencies?

---

Update 1: A use case: Suppose you have a lot of published documents or a large manuscript, and want to be sure that their compiled versions haven't changed due to a package update. It may be the case that you want to change the output, hence the reason for a package update, but it's most important that you know if anything has changed, and you can subsequently discern how and why they changed.

Answer 1

The LaTeX3 Project has some experience in this area, although most of our automated tests are currently focussed more on code input/output than typeset material. These scripts were developed for LaTeX2e, but have been somewhat updated for work on LaTeX3.

In the context of working with boxes and the output routine, we do have two forms of regression testing that do test typeset material, at least to some extent. Both these and the more programmatic tests rely on using one or more test documents which write verbose information to the log file. After some script-based cleanup to remove 'dynamic' material, these can then be compared with a reference version of the same log, to check the input and output are the same.

To construct appropriate test files, some imagination may be needed. For example, for box-based test we use something akin to \showthe\box..., which will show the content of a box in the log. Thus if you box up the material to be checked, then write the box to the log, you can see exactly what is there.

For full pages, you can use \tracingall during shipout to record everything on a page in the log. Complementing this, we also have 'trace' DocStrip guards in, for example, the output routine code. This extra code writes very detailed information to the log, for example every outcome of a calculation. Change to the layout will alter these numbers, if only subtly, and this can then be picked up by a script.

The implementation that the LaTeX3 Project use for testing is all available from our GitHub site. The key mechanisms are a special LaTeX file which is used by the typesetting tests, and a small Perl script which formats the logs (at some stage, it would be nice to rewrite the latter in Lua, as it would keep things more self-contained). Supporting these two files are a number of Makefile/batch scripts, all of the same general form, and the various test files themselves (test input as .lvt, test output as .tlg).

Update September 2016

The team have worked hard on testing, and have developed l3build as an integrated testing-and-building system for (La)TeX. This can also be used in automated testing, which the team have set up using Travis-CI to carry this out on each commit. There is a TUGboat article on l3build and one on our Travis set up. The latter is (currently) only available to members direct from TUG, but you can get the source from the LaTeX3 GitHub repo.

Answer 2

This is not really a complete answer, just some thoughts and hints. On the other hand, it is too long for a comment.

So given that we have updated a couple of packages:

• A first regression could be to compile all test documents and see if there is any diff in the compilation process itself by comparing the logfiles. New compilation errors do obviously indicate a regression problem, but new warnings might also be helpful. Of course, the logged output by some package might have changed without any "substantial change" to the package itself. However, grep`ing for "magic terms", such as "depricated" might nevertheless trigger potential regression issues.
• Given that the documents still compile, the important point is, of course, the visual difference. An obvious approach is to compare the resulting PDFs, page by page. Unfortunately, very few "PDF diff tools" actually compute a visual diff. AFAIK, Adobe Acrobat Pro can do this. A bit more scripting-friendly (even though Acrobat is said to be well scriptable) might be a solution that involves ImageMagick or some other converter to first convert each PDF page-wise into PNG graphics. The PNG pages (in theory) should then be binary "diffable". At least one should be able to find pages with some visual difference.
• Depending on your set of "test documents", the previous point might involve a high computational overhead. If you have a couple of books that, all together, make up some thousand pages, a blind page-wise comparison of everything will be expensive. It might be a good idea to use the -recorder option of pdflatex to compute for each document a list of the actually used input files (including packages and all their dependencies) and to use this information to do the regression only with those documents that actually depend on a changed package.