What's the folk-lore on automatic testing of TeX programming?

Question

Say you write a macro or a package, which goes through the usual course of software life-cycle. How would you define a set of tests, and have this set run automatically to make sure that changes to the code are in compatible, in some sense of the word?

Answer 1

I think there are three broad categories for unit testing in TeX based on:

1. Log file output
2. DVI/PDF generated output
3. TeX-based programming checks

Log file output

Joseph has already mentioned the log-file-based testing used for LaTeX2e and for expl3, but it's worth noting that these tests can also be used for checking the contents of boxes and output, so they're quite general in their scope. For example:

\tracingoutput=1 \tracingonline=1
\showboxdepth=999 \showboxbreadth=999

\newbox\foo
\savebox\foo{abc $\cos(x)$}
\showbox\foo

Generated output

For unicode-math and fontspec I've got a different system that is slower and less reliable than log file tests, in which I use generated PDFs to compare before/after differences using ImageMagick's compare tool.

One of the main problems with this approach is the presence of false positives due to off-by-one pixel rounding errors in the rasterisation. (And, possibly, installing new fonts changing the outputs of the tests.)

On thing I like about comparing PDFs, though, is that I've then got a useable document based on the test suite for demonstration purposes. No reason this couldn't be done as a by-product of any of the other testing processes, however.

Using a DVI-based comparison as suggested by other commenters may be more reliable, but I haven't looked into the various options there.

Programming checks

Both of the above techniques require Makefile or other scripts to automate. An easier method, albeit less flexible, is to use the qstest package to write unit tests as part of a TeX document (often included in the dtx source file). We've used this successfully in the hardwrap package, but this technique does require that the test can somehow be checked from within TeX — and often this is sufficient.

As an example from hardwrap.dtx, this sort of test looks like

\begin{qstest}{deal with explicit newlines}{}
\HardWrap{\xdef\TMP}{50}{}{NEWLINE}{aaa bbb^^Jccc}%
\Expect *{\TMP} *{aaa bbbNEWLINEccc}
\end{qstest}

Answer 2

If you're adding new features, and you have a collection of LaTeX (or TeX) files that don't use the new features, you can check that none of those files are affected by the change by using dvii, from the dvii homepage. If you use the command dvii -p -M1 myfile.dvi > myfile.digest, you'll get a "message digest", i.e., a list of "hashes", one for each page. If you then recreate myfile.dvi using your changed version, you can compare the digests to quickly see if anything on any of the pages has changed.

For example, I have a large collection of LaTeX files that I use for testing a document class. I have a script makedigest as follows:

#! /bin/sh

# We assume that the argument is the name of a dvi file.
# We write the message digest to standard output.

dvii -p -M1 $*

and for each file myfile.tex I've run makedigest to create myfile.dvi.digest.old. When I make a change that's not supposed to affect any of those files, I run the script checknew, which consists of

#!/bin/sh

for f in *tex;do
  for i in `seq 1 6`;do
    latex $f
  done
done

for f in *.dvi;do
  makedigest $f > ${f}.digest.new
done

echo

echo "Checking for errors:"
grep rror *.log
echo "Checking for warnings:"
grep Warning *.log

echo "Comparing new digests with old:"
for f in *.dvi;do
  echo "${f}:"
  diff ${f}.digest.old ${f}.digest.new
done

This quickly runs LaTeX enough times to get a stable version, checks for errors or warnings, and compares the new digests with the old. Thus, I quickly see which pages of which test documents have been changed unexpectedly.

Answer 3

For both the LaTeX2e kernel and the current LaTeX3 code, there are a long series of automated tests which use log data only. These test things at the programming end of the spectrum, where you can write the results to the log. A bit of post-processing results in a file containing only the things you need to test and not the variable stuff (for example, which \count is being used for a particular variable). The biggest problem is then not so much the testing itself as writing the tests in the first place. (The automated system for LaTeX3 is available from the public SVN: http://www.latex-project.org/svnroot/experimental/trunk/)

Answer 4

I've been wondering about writing unit tests for my LaTeX packages.

For now, what I'm doing is testing the produced PDFs. I just started today doing this using rspec and pdf-reader` in ruby.

The moderntimeline package now has unit tests based on the produced PDF. The unit tests are in the spec/pdf_spec.rb file, and so far it mostly checks a few things such as the number of pages, the creator, the media box coordinates, the number of fonts and some of the contents.

I've integrated the tests into Travis CI to achieve continuous integration of the package. See the .travis.yml file in the repository for how this is done.