# How to do unit test for graphics generated by evolving packages?

Assume we have a lot of diagrams generated by one of any existing packages (PSTricks, TikZ, Asymptote, etc). We know that the packages are evolving and we need to make sure our existing codes are still correct without having to check it one by one visually.

Of course the package authors might do check with their own test code. As the code output is an image and the author's test code might not be enough to test every possible cases, then we need to do unit test ourselves. How to do this? Using image processing?

• pgfplots compares pixels of the generated graphics using ImageMagick. This is pretty fragile though and of course breaks when you swap out fonts or colors. Another approach is to diff the output of \showlists which has the advantage that it is pure ASCII text. That's what the latex3 project is doing and is what I want to do for PGF/TikZ. Jun 18 '20 at 1:53
• @HenriMenke: How can \showlists be used to check the visual appearance of graphics code? Jun 18 '20 at 2:11
• Jun 18 '20 at 2:52
• @MoneyOrientedProgrammer You make sure you get the same output sent to the PDF/DVI file: often, package changes are about syntax, correcting erroneous paths, adding features, not changing the result of existing input Jun 18 '20 at 5:25
• @JosephWright Thank you.... Jun 18 '20 at 5:32

There are several approaches that you can take to regression-test your graphics.

1. Pixel comparison of the output file
2. Comparison of log files
3. Textual comparison of the PDF files

# 1. Pixel comparison of the output file

This approach basically boils down to “know your tools”. Here I will use the ImageMagick tool compare to compare the pixels of two pictures. Let's start from the MWE

\documentclass{standalone}
\usepackage{tikz}
\begin{document}
\begin{tikzpicture}
\draw (0,0) -- (1,1);
\end{tikzpicture}
\end{document}


and typeset this with

pdflatex test.tex
mv test.pdf test.ref.pdf


This is our reference file. Then later we can compare a new version to the reference file using

compare test.pdf test.ref.pdf -metric MAE diff.png
echo $?  This might print some warnings regarding color profiles but only the exit status ($?) is really important, where 0 means success and 1 means failure and anything else means that there was an error in the command you typed. If we now change the end point of the line from (1,1) to (1.1,1.1) the output will on the terminal will be

1203.18 (0.0183594)1


The trailing 1 is the exit status (fail) and diff.png will now contain an image which highlights difference.

As you can see, in the default settings this is microscopically small. Figuring out how to get a higher resolution is left as an exercise to the reader (Hint: Consult the manual).

• Works with any image and is not specific to TeX.
• Works across engines, i.e. you could compare the reference picture for pdfTeX with the output of LuaTeX to make sure both generate the same picture.

• Requires lots of storage. You have to store the full output file for each input.
• Even though it works across engine in principle, differences in font metrics make this very fragile and not generally usable.
• ImageMagick doesn't really have a great track record in terms of security (https://imagetragick.com/). This might be not so much of a concern here because after all we can trust the input file because we generated it, but if this were used to roll out a unit testing system where other can submit input, I'd be more careful.

# 2. Comparison of log files

This approach is also taken by the l3build system, which I'm sure provides a much better interface, but is sort of limited to the “classic LaTeX package” packaging workflow.

This uses the fact that graphics are implemented in terms of device-dependent \special instructions (or \pdfliteral etc.). These will show up in the log file when \tracingoutput is active. It only requires a little instrumentation around the picture itself.

\documentclass{article}
\usepackage{tikz}
\begin{document}
\showboxdepth=\maxdimen
\tracingoutput=1
\tracingonline=1
\typeout{START}
\shipout\vbox{
\begin{tikzpicture}
\draw (0,0) -- (1,1);
\end{tikzpicture}
}
\typeout{END}
\tracingoutput=0
\tracingonline=0
\end{document}


The interesting bit of the log file is the one that contains the content of the START...END bracket. We don't really care about anything else, because the test should be agnostic to package versions and other “impure” things like the time of day. In this instance I typeset the example with

pdflatex test.tex


You can extract the relevant bit with any processing and reporting language, I used

awk '/^START$/,/^END$/ { print }' test.log > test.ref.log


Just save that along your image source and use, e.g. the standard POSIX tool diff to compare the output.

START

Completed box being shipped out [1]
\vbox(28.85274+0.0)x345.0
.\hbox(0.0+0.0)x0.0
.\hbox(28.85274+0.0)x345.0, glue set 301.14726fil
..\hbox(0.0+0.0)x15.0
..\hbox(28.85274+0.0)x28.85274
...\glue 0.2
...\hbox(0.0+0.0)x0.0, shifted -0.2
....\pdfliteral{q }
....\pdfliteral{0 G }
....\pdfliteral{0 g }
....\pdfliteral{0.3985 w }
....\hbox(0.0+0.0)x0.0
.....\pdfliteral{q }
.....\glue 0.0
.....\pdfliteral{0.0 0.0 m }
.....\pdfliteral{28.3468 28.3468 l }
.....\pdfliteral{S }
.....\glue 0.0
.....\glue 0.0
.....\pdfliteral{Q }
.....\glue 0.0 plus 1.0fil minus 1.0fil
....\pdfliteral{n }
....\pdfliteral{Q }
....\glue 0.0 plus 1.0fil minus 1.0fil
..\penalty 10000
..\glue(\parfillskip) 0.0 plus 1.0fil
..\glue(\rightskip) 0.0

END


Then you can later compare this output using

pdflatex test.tex
awk '/^START$/,/^END$/ { print }' test.log | diff -u - test.ref.log


If I change for example the end point of the line from (1,1) to (1.1,1.1) the diff will show

--- -   2020-06-18 14:12:20.880605535 +1200
+++ test.ref.log        2020-06-18 14:10:56.947754945 +1200
@@ -1,11 +1,11 @@
START

Completed box being shipped out [1]
-\vbox(31.69818+0.0)x345.0
+\vbox(28.85274+0.0)x345.0
.\hbox(0.0+0.0)x0.0
-.\hbox(31.69818+0.0)x345.0, glue set 298.30182fil
+.\hbox(28.85274+0.0)x345.0, glue set 301.14726fil
..\hbox(0.0+0.0)x15.0
-..\hbox(31.69818+0.0)x31.69818
+..\hbox(28.85274+0.0)x28.85274
...\glue 0.2
...\hbox(0.0+0.0)x0.0, shifted -0.2
....\pdfliteral{q }
@@ -16,7 +16,7 @@
.....\pdfliteral{q }
.....\glue 0.0
.....\pdfliteral{0.0 0.0 m }
-.....\pdfliteral{31.18166 31.18166 l }
+.....\pdfliteral{28.3468 28.3468 l }
.....\pdfliteral{S }
.....\glue 0.0
.....\glue 0.0


• Requires only very little storage and the reference data is plain text (very version control friendly).
• Generation of and comparison with reference data is straight-forward and needs only very little tooling.
• Tracking down what changed might be a bit easier.

• Doesn't work outside of TeX.
• Doesn't work across engines
• Is susceptible to changes in the TeX engine itself which don't actually change the visible output.
• This naive approach doesn't account for global PDF objects like patterns. A pattern will only show up as a reference to the pattern \pdfliteral{/pgfprgb cs 0 0 0 /pgfpat3 scn } but the pattern code itself is not shown.

# 3. Textual comparison of the PDF files

Another approach that you can take is perform a textual comparison of the entire PDF. To this end you will have to remove some of the state that TeX inserts into the PDF like the name and version of the engine or the time of day. Additionally I turn off the PDF compression such that a textual comparison actually yields usable output.

\pdfcompresslevel=0
\pdfobjcompresslevel=0
\pdfinfo{/Producer (pdfTeX)}
\pdfinfoomitdate=1
\pdfsuppressptexinfo=1
\pdftrailerid{}
\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{patterns}
\begin{document}
\begin{tikzpicture}
\draw[pattern=north east lines] (0,0) rectangle (1,1);
\end{tikzpicture}
\end{document}


This is all very specific to pdfTeX. For other engines the primitives are very different and for XeTeX you maybe want to compare the XDV files, depending on what you are after. To generate the reference data simply typeset the document

pdflatex test.tex
mv test.pdf test.ref.pdf


Now we change north east lines to north west lines in the example. This change can also be caught by the previous approach because the two patterns have different ids but if the pattern itself changes without changing its id, this will go unnoticed. After typesetting the changed example we can run diff

diff -au test.pdf test.ref.pdf


The -a flag tells the GNU implementation of diff to treat the input as text even if binary data is detected. Depending on which system you run the flag might have a different name. The resulting difference is a bit unwieldy but we can see that the pattern itself has changed (lines starting with +/Length and +q) and some xref stuff that we don't really care about.

--- test.pdf    2020-06-18 16:34:54.345842013 +1200
+++ test.ref.pdf        2020-06-18 16:24:34.586572786 +1200
@@ -3,10 +3,10 @@
4 0 obj
<<
/Type /Pattern /PatternType 1 /PaintType 2 /TilingType 1 /BBox [-0.99628 -0.99628 3.9851 3.9851] /XStep 2.98883 /YStep 2.98883 /Matrix [1.0 0.0 0.0 1.0 0.0 0.0] /Resources << >>
-/Length 48
+/Length 43
>>
stream
-q 0.3985 w 0.0 2.98883 m 3.08846 -0.09962 l S Q
+q 0.3985 w 0.0 0.0 m 3.08846 3.08846 l S Q
endstream
endobj
7 0 obj
@@ -24,7 +24,7 @@
0.3985 w
q
q
-/pgfprgb cs 0 0 0 /pgfpat4 scn
+/pgfprgb cs 0 0 0 /pgfpat3 scn
0.0 0.0 m
0.0 0.0 m
0.0 28.3468 l
@@ -66,7 +66,7 @@
<<>>
endobj
2 0 obj
-<< /pgfpat4 4 0 R>>
+<< /pgfpat3 4 0 R>>
endobj
3 0 obj
<<  /pgfprgb [/Pattern /DeviceRGB] >>
@@ -200,25 +200,25 @@
xref
0 15
0000000000 65535 f
-0000000893 00000 n
-0000000913 00000 n
-0000000948 00000 n
+0000000888 00000 n
+0000000908 00000 n
+0000000943 00000 n
0000000015 00000 n
-0000000774 00000 n
-0000000670 00000 n
-0000000301 00000 n
-0000008648 00000 n
-0000008786 00000 n
-0000001001 00000 n
-0000001023 00000 n
-0000008429 00000 n
-0000008843 00000 n
-0000008893 00000 n
+0000000769 00000 n
+0000000665 00000 n
+0000000296 00000 n
+0000008643 00000 n
+0000008781 00000 n
+0000000996 00000 n
+0000001018 00000 n
+0000008424 00000 n
+0000008838 00000 n
+0000008888 00000 n
trailer
<< /Size 15
/Root 13 0 R
/Info 14 0 R
>>
startxref
-8965
+8960
%%EOF


This textual comparison is somewhat useful thanks to the fact that we have turned off the PDF compression earlier. In principle we could also leave the compression enable and drop the -a flag from diff in which case the output would simply be

Binary files test.pdf and test.ref.pdf differ


This might or might not be what you want.