# What's the folk-lore on automatic testing of TeX programming?

Say you write a macro or a package, which goes through the usual course of software life-cycle. How would you define a set of tests, and have this set run automatically to make sure that changes to the code are in compatible, in some sense of the word?

-
For svn-multi I made several (~50) test bench files which I can run automatically using a Makefile. You can have a look at them with the SVN browser of my website. ATM the Makefile only tests for compile errors. Some of the test files compare the results and cause such an error on purpose if the result isn't as expected. I wanted to create a real test package which a lot of macros to test packages, but never found the time. –  Martin Scharrer Feb 25 '11 at 17:47
It seems not easy to write a simple regression test for TeX, since the test result are often typesetting output. PDF format is not reproducible, DVI may be better but I never tried. Mostly testing results need to be examined by human being. –  Leo Liu Feb 25 '11 at 18:03
Generate dvi and then use dvitype + diff to validate the output. –  lhf Feb 25 '11 at 18:08
It depends what you want to test. It is easier if the package produces something which can be compared as string or using lengths, e.g. I test if any command generate unwanted spaces by placing them inside a box which should be 0x0x0). If it is graphical output you need to compare it using image comparison software. I did something like that for the tikz-timing package to check if the timing transients are still correct. –  Martin Scharrer Feb 25 '11 at 18:18

I think there are three broad categories for unit testing in TeX based on:

1. Log file output
2. DVI/PDF generated output
3. TeX-based programming checks

## Log file output

Joseph has already mentioned the log-file-based testing used for LaTeX2e and for expl3, but it's worth noting that these tests can also be used for checking the contents of boxes and output, so they're quite general in their scope. For example:

\tracingoutput=1 \tracingonline=1

\newbox\foo
\savebox\foo{abc $\cos(x)$}
\showbox\foo


## Generated output

For unicode-math and fontspec I've got a different system that is slower and less reliable than log file tests, in which I use generated PDFs to compare before/after differences using ImageMagick's compare tool.

One of the main problems with this approach is the presence of false positives due to off-by-one pixel rounding errors in the rasterisation. (And, possibly, installing new fonts changing the outputs of the tests.)

On thing I like about comparing PDFs, though, is that I've then got a useable document based on the test suite for demonstration purposes. No reason this couldn't be done as a by-product of any of the other testing processes, however.

Using a DVI-based comparison as suggested by other commenters may be more reliable, but I haven't looked into the various options there.

## Programming checks

Both of the above techniques require Makefile or other scripts to automate. An easier method, albeit less flexible, is to use the qstest package to write unit tests as part of a TeX document (often included in the dtx source file). We've used this successfully in the hardwrap package, but this technique does require that the test can somehow be checked from within TeX — and often this is sufficient.

As an example from hardwrap.dtx, this sort of test looks like

\begin{qstest}{deal with explicit newlines}{}
\HardWrap{\xdef\TMP}{50}{}{NEWLINE}{aaa bbb^^Jccc}%
\Expect *{\TMP} *{aaa bbbNEWLINEccc}
\end{qstest}

-
I also found a unittest framework following 2.: pdf generated output quite usable. I liked it to work with the compare tool of imagemagick; and I combined it with git such I can always checkout a reference revision of my code, generate the reference pdf, then I compare the actual against that reference. However, all these scripts require quite some effort until everythings works as expected. –  Christian Feuersänger Apr 28 '11 at 18:49

If you're adding new features, and you have a collection of LaTeX (or TeX) files that don't use the new features, you can check that none of those files are affected by the change by using dvii, from the dvii homepage. If you use the command dvii -p -M1 myfile.dvi > myfile.digest, you'll get a "message digest", i.e., a list of "hashes", one for each page. If you then recreate myfile.dvi using your changed version, you can compare the digests to quickly see if anything on any of the pages has changed.

For example, I have a large collection of LaTeX files that I use for testing a document class. I have a script makedigest as follows:

#! /bin/sh

# We assume that the argument is the name of a dvi file.
# We write the message digest to standard output.

dvii -p -M1 $*  and for each file myfile.tex I've run makedigest to create myfile.dvi.digest.old. When I make a change that's not supposed to affect any of those files, I run the script checknew, which consists of #!/bin/sh for f in *tex;do for i in seq 1 6;do latex$f
done
done

for f in *.dvi;do
makedigest $f >${f}.digest.new
done

echo

echo "Checking for errors:"
grep rror *.log
echo "Checking for warnings:"
grep Warning *.log

echo "Comparing new digests with old:"
for f in *.dvi;do
echo "${f}:" diff${f}.digest.old \${f}.digest.new
done


This quickly runs LaTeX enough times to get a stable version, checks for errors or warnings, and compares the new digests with the old. Thus, I quickly see which pages of which test documents have been changed unexpectedly.

-
Just a tip: for non-TeX source code the <pre> .. </pre> HTML tags are better to avoid TeX syntax highlighting. In your post this isn't a big deal, thought. –  Martin Scharrer Feb 25 '11 at 18:43
@Martin Thanks, but: What sort of content is supposed to be enclosed by <pre>..</pre>? Even better: Where can I find a description of how to properly mark things up when posting here? (I never remember how to mark clickable links here.) Thanks –  Phil Hirschhorn Feb 25 '11 at 18:51
Here the shell scripts should be enclosed in <pre> tags instead of indenting them with four spaces. See the editing or the help button just below the Add Comment button beside the comment text field, for help on the available syntax. There is also this post by Hendrik Vogt which talks a lot about code highlighting. –  Martin Scharrer Feb 25 '11 at 18:56

For both the LaTeX2e kernel and the current LaTeX3 code, there are a long series of automated tests which use log data only. These test things at the programming end of the spectrum, where you can write the results to the log. A bit of post-processing results in a file containing only the things you need to test and not the variable stuff (for example, which \count is being used for a particular variable). The biggest problem is then not so much the testing itself as writing the tests in the first place. (The automated system for LaTeX3 is available from the public SVN: http://www.latex-project.org/svnroot/experimental/trunk/)

-
For now, what I'm doing is testing the produced PDFs. I just started today doing this using rspec and pdf-reader in ruby.
The moderntimeline package now has unit tests based on the produced PDF. The unit tests are in the spec/pdf_spec.rb file, and so far it mostly checks a few things such as the number of pages, the creator, the media box coordinates, the number of fonts and some of the contents.
I've integrated the tests into Travis CI to achieve continuous integration of the package. See the .travis.yml` file in the repository for how this is done.