Say you write a macro or a package, which goes through the usual course of software life-cycle. How would you define a set of tests, and have this set run automatically to make sure that changes to the code are in compatible, in some sense of the word?
4 Answers
I think there are three broad categories for unit testing in TeX based on:
- Log file output
- DVI/PDF generated output
- TeX-based programming checks
Log file output
Joseph has already mentioned the log-file-based testing used for LaTeX2e and for expl3, but it's worth noting that these tests can also be used for checking the contents of boxes and output, so they're quite general in their scope. For example:
\tracingoutput=1 \tracingonline=1
\showboxdepth=999 \showboxbreadth=999
\newbox\foo
\savebox\foo{abc $\cos(x)$}
\showbox\foo
Generated output
For unicode-math
and fontspec
I've got a different system that is slower and less reliable than log file tests, in which I use generated PDFs to compare before/after differences using ImageMagick's compare
tool.
One of the main problems with this approach is the presence of false positives due to off-by-one pixel rounding errors in the rasterisation. (And, possibly, installing new fonts changing the outputs of the tests.)
On thing I like about comparing PDFs, though, is that I've then got a useable document based on the test suite for demonstration purposes. No reason this couldn't be done as a by-product of any of the other testing processes, however.
Using a DVI-based comparison as suggested by other commenters may be more reliable, but I haven't looked into the various options there.
Programming checks
Both of the above techniques require Makefile
or other scripts to automate. An easier method, albeit less flexible, is to use the qstest
package to write unit tests as part of a TeX document (often included in the dtx
source file). We've used this successfully in the hardwrap
package, but this technique does require that the test can somehow be checked from within TeX — and often this is sufficient.
As an example from hardwrap.dtx
, this sort of test looks like
\begin{qstest}{deal with explicit newlines}{}
\HardWrap{\xdef\TMP}{50}{}{NEWLINE}{aaa bbb^^Jccc}%
\Expect *{\TMP} *{aaa bbbNEWLINEccc}
\end{qstest}
-
1I also found a unittest framework following 2.: pdf generated output quite usable. I liked it to work with the
compare
tool of imagemagick; and I combined it with git such I can always checkout a reference revision of my code, generate the reference pdf, then I compare the actual against that reference. However, all these scripts require quite some effort until everythings works as expected. Apr 28, 2011 at 18:49
If you're adding new features, and you have a collection of LaTeX (or TeX) files that don't use the new features, you can check that none of those files are affected by the change by using dvii
, from the dvii homepage. If you use the command dvii -p -M1 myfile.dvi > myfile.digest
, you'll get a "message digest", i.e., a list of "hashes", one for each page. If you then recreate myfile.dvi using your changed version, you can compare the digests to quickly see if anything on any of the pages has changed.
For example, I have a large collection of LaTeX files that I use for testing a document class. I have a script makedigest
as follows:
#! /bin/sh # We assume that the argument is the name of a dvi file. # We write the message digest to standard output. dvii -p -M1 $*
and for each file myfile.tex I've run makedigest
to create myfile.dvi.digest.old
. When I make a change that's not supposed to affect any of those files, I run the script checknew
, which consists of
#!/bin/sh for f in *tex;do for i in `seq 1 6`;do latex $f done done for f in *.dvi;do makedigest $f > ${f}.digest.new done echo echo "Checking for errors:" grep rror *.log echo "Checking for warnings:" grep Warning *.log echo "Comparing new digests with old:" for f in *.dvi;do echo "${f}:" diff ${f}.digest.old ${f}.digest.new done
This quickly runs LaTeX enough times to get a stable version, checks for errors or warnings, and compares the new digests with the old. Thus, I quickly see which pages of which test documents have been changed unexpectedly.
-
1Just a tip: for non-TeX source code the
<pre> .. </pre>
HTML tags are better to avoid TeX syntax highlighting. In your post this isn't a big deal, thought. Feb 25, 2011 at 18:43 -
@Martin Thanks, but: What sort of content is supposed to be enclosed by
<pre>..</pre>
? Even better: Where can I find a description of how to properly mark things up when posting here? (I never remember how to mark clickable links here.) Thanks Feb 25, 2011 at 18:51 -
1Here the shell scripts should be enclosed in
<pre>
tags instead of indenting them with four spaces. See the editing or thehelp
button just below theAdd Comment
button beside the comment text field, for help on the available syntax. There is also this post by Hendrik Vogt which talks a lot about code highlighting. Feb 25, 2011 at 18:56
For both the LaTeX2e kernel and the current LaTeX3 code, there are a long series of automated tests which use log data only. These test things at the programming end of the spectrum, where you can write the results to the log. A bit of post-processing results in a file containing only the things you need to test and not the variable stuff (for example, which \count
is being used for a particular variable). The biggest problem is then not so much the testing itself as writing the tests in the first place. (The automated system for LaTeX3 is available from the public SVN: http://www.latex-project.org/svnroot/experimental/trunk/)
I've been wondering about writing unit tests for my LaTeX packages.
For now, what I'm doing is testing the produced PDFs. I just started today doing this using rspec
and pdf-reader
` in ruby.
The moderntimeline package now has unit tests based on the produced PDF. The unit tests are in the spec/pdf_spec.rb
file, and so far it mostly checks a few things such as the number of pages, the creator, the media box coordinates, the number of fonts and some of the contents.
I've integrated the tests into Travis CI to achieve continuous integration of the package. See the .travis.yml
file in the repository for how this is done.
svn-multi
I made several (~50) test bench files which I can run automatically using a Makefile. You can have a look at them with the SVN browser of my website. ATM the Makefile only tests for compile errors. Some of the test files compare the results and cause such an error on purpose if the result isn't as expected. I wanted to create a real test package which a lot of macros to test packages, but never found the time.tikz-timing
package to check if the timing transients are still correct.l3build
looks promising.