How to get pdftex to have same PDF output for each run on the same input

Question

Is there a way to get pdftex to output the exact same pdf each time it is run on the same input?

We are using an in-house tool to generate LaTeX for engineering design documentation. We run the resulting .tex file through pdflatex (which is linked to pdftex) on a Linux system to generate the documentation

$ pdflatex -version
pdfTeX 3.14159265-2.6-1.40.15 (TeX Live 2014)
kpathsea version 6.2.0
Copyright 2014 Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX).
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX).
Compiled with libpng 1.6.10; using libpng 1.6.10
Compiled with zlib 1.2.8; using zlib 1.2.8
Compiled with xpdf version 3.03

This works, except that each time pdftex is run on the same input file it generates a slightly different .pdf. The differences are not apparent when the files are viewed or even when they are run through a couple of the online pdf diff tools, but regular old diff says they are different, and the sha512 hashes are different:

$ ls
foo.tex
$ pdflatex foo.tex
This is pdfTeX, Version 3.14159265-2.6-1.40.15 (TeX Live 2014) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode

<A bunch of noise>

Output written on foo.pdf (29 pages, 167416 bytes).
Transcript written on foo.log.
$ mv foo.pdf first.foo.pdf
$ pdflatex foo.tex
This is pdfTeX, Version 3.14159265-2.6-1.40.15 (TeX Live 2014) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode

<More noise>

Output written on foo.pdf (29 pages, 167416 bytes).
Transcript written on foo.log.
$ diff first.foo.pdf foo.pdf 
Binary files first.foo.pdf and foo.pdf differ
$ sha512sum -b first.foo.pdf 
cdfe293ef87e0ef6721fb749ea964bbeb88438df382302ff4b767e51e63e2831423ae960ceb0d2dc71357125efe906b349960f525d8b73a997c9bcf8e356d454 *first.foo.pdf
$ sha512sum -b foo.pdf 
81def851fbaa9aafa79ce524215916d455a5ed4d7062b639900793c3ceeea73992924462baa479f8fd655d0c3f8130a85c9cae0186b0e764c343f11d799c9b1e *foo.pdf

Using hexdump to diff the files, it looks suspiciously like the first two differences are datestamps:

$ hexdump -C first.foo.pdf > first.foo.pdf.hexdump
$ hexdump -C foo.pdf > foo.pdf.hexdump
$ diff foo.pdf.hexdump first.foo.pdf.hexdump 
10241c10241
< 00028000  31 38 30 37 31 31 32 33  35 31 31 37 5a 29 0a 2f  |180711235117Z)./|
---
> 00028000  31 38 30 37 31 31 32 33  35 31 30 31 5a 29 0a 2f  |180711235101Z)./|
10243c10243
< 00028020  37 31 31 32 33 35 31 31  37 5a 29 0a 2f 54 72 61  |711235117Z)./Tra|
---
> 00028020  37 31 31 32 33 35 31 30  31 5a 29 0a 2f 54 72 61  |711235101Z)./Tra|
10431,10435c10431,10435
< 00028be0  20 30 20 52 0a 2f 49 44  20 5b 3c 33 35 34 33 35  | 0 R./ID [<35435|
< 00028bf0  45 35 35 30 32 38 36 30  45 39 45 39 31 44 31 41  |E5502860E9E91D1A|
< 00028c00  38 30 32 43 45 30 34 36  36 44 39 3e 20 3c 33 35  |802CE0466D9> <35|
< 00028c10  34 33 35 45 35 35 30 32  38 36 30 45 39 45 39 31  |435E5502860E9E91|
< 00028c20  44 31 41 38 30 32 43 45  30 34 36 36 44 39 3e 5d  |D1A802CE0466D9>]|
---
> 00028be0  20 30 20 52 0a 2f 49 44  20 5b 3c 46 39 33 31 36  | 0 R./ID [<F9316|
> 00028bf0  42 39 46 42 41 36 37 37  46 41 35 32 35 36 46 39  |B9FBA677FA5256F9|
> 00028c00  37 42 43 37 33 38 45 32  38 43 34 3e 20 3c 46 39  |7BC738E28C4> <F9|
> 00028c10  33 31 36 42 39 46 42 41  36 37 37 46 41 35 32 35  |316B9FBA677FA525|
> 00028c20  36 46 39 37 42 43 37 33  38 45 32 38 43 34 3e 5d  |6F97BC738E28C4>]|

And I suspect the subsequent differences are checksums/hashes which I would expect to be different with different timestamps.

None of this would really be a problem, except that for reasons beyond the scope of this question, we regenerate the pdf frequently and commit the generated pdf files into git (using LFS) and because of the differences, every time we do a commit git thinks the pdf has been changed.

in its -help, pdftex makes an oblique reference to changing the output comment from the date to an arbitrary string, but only in dvi mode. I tried anyway and it didn't resolve the issue.

Is there a way to get pdftex to output the exact same pdf each time it is run on the same input? Or, lacking that, is there a way to modify the pdf to use an arbitrary timestamp?

Answer 1

This was an interesting question :)

The pdfTeX manual shows some concerns with reproducible PDFs.

There are two (three) primitives to set if you want every run of pdfTeX to generate the exact same PDF.

First there is /CreationDate and /ModDate. You can either set these values with \pdfinfo:

\pdfinfo{%
  /CreationDate D:YYYYMMDDhhmmssTZ
  /ModDate      D:YYYYMMDDhhmmssTZ
}

or you can completely disable them with \pdfinfoomitdate=1.

Then there is the ID you mentioned in your question. You can set the /ID with the \pdftrailerid primitive. You can give it any text, including an empty string: \pdftrailerid{}.

These two will guarantee that two runs of pdfTeX in the same computer with the same version of everything will generate the same PDF.

There is another set of data recorded that depends on the version of pdfTeX, and the TeX distribution you are using, so this may not hold true if, for example, you update your system. There are four more recorded items:

1     PTEX.Fullbanner
2     PTEX.FileName
4     PTEX.PageNumber
8     PTEX.InfoDict

these can be disabled selectively by passing the sum of the ones you want to disable to \pdfsuppressptexinfo. For example, \pdfsuppressptexinfo=3 will supress the Fullbanner and the FileName. To disable all you can pass \pdfsuppressptexinfo=-1.

So, in conclusion, the following code will generate the same PDF in every run of pdfTeX:

\pdfinfoomitdate1
\pdfsuppressptexinfo-1
\pdftrailerid{}

\input story

\bye

---

Update:

As John DeRoo said, he had to update his system to get it working with the solution above. Reading a little bit further pdfTeX's manual, one can see it says that "the primitive was introduced in pdfTeX 1.40.17". In fact, the NEWS file contains:

pdfTeX 3.14159265-2.6-1.40.17 (TeX Live 2016)  (May 20, 2016)
- changes:
  - if the environment variable SOURCE_DATE_EPOCH is set, use its value for
    the PDF CreationDate and ModDate values, and to seed the trailer /ID.
    This by itself should suffice to create reproducible PDFs.  The
    new primitives below support more granular output tweaks with the
    same intent.
  - if the environment variable SOURCE_DATE_EPOCH_TEX_PRIMITIVES is set
    to 1, the \year, \day, and \time primitives are also initialized
    from the SOURCE_DATE_EPOCH value, instead of the current time.

  - new primitive \pdfinfoomitdate to omit CreationDate and ModDate keys.
  - new primitive \pdftrailerid to set seed for the trailer /ID
    computation; with an empty argument \pdftrailerid{}, the /ID is omitted.
  - new primitive \pdfsuppressptexinfo to omit PTEX.* keys from output;
    the value is a bitmask:
    % 1 -> PTEX.Fullbanner
    % 2 -> PTEX.FileName
    % 4 -> PTEX.PageNumber
    % 8 -> PTEX.InfoDict (/Producer /Creator /CreationDate /ModDate /Trapped)

so in versions prior to May 2016 reproducible PDFs aren't possible. The /CreationDate and /ModDate can still be changed with \pdfinfo but the /ID is still different.