# Why does xelatex produce different files from the same deterministic sources?

Take the simple latex source file:

\documentclass{article}
\usepackage{lipsum}
\usepackage{amsgen}
\begin{document}
\lipsum[1-10]
\end{document}


When processed with "latex" you always get the same DVI file, identical everywhere except for the date.

When processed (multiple times) with "pdflatex" you get the same PDF file, with the exception of an ID and the Date, and the same thing for "lualatex".

But when processed (multiple times) with "xelatex" you get wildly different PDF files, with different sizes. One can see the differences easily with "vimdiff".

Why is it the processing with "xelatex" not deterministic - not the same for the same sources?

• I get no visual changes with xelatex. The pdf filesize oscillates between 10727, 10728, or 10729 bytes. Thus "wildly different" definitely does not apply on my system (mac os x). Which OS do you use? From which editor do you compile the source? – user4686 Feb 13 '16 at 7:22
• vimdiff indeed indicates differences inside the pdf. Perhaps something having to do with font compression. Needs a pdf expert here. – user4686 Feb 13 '16 at 7:33
• XeTeX produces exactly the same xdv files on multiple runs, i.e. their TeX contents are the same. Where's the issue here? (xdvipdfmx and dvipdfmx both produce some variation in the find PDF file but as already commented this is likely to do with font compression.) – Joseph Wright Feb 13 '16 at 8:12
• @AFeldman: That's wrong: luatex does not claim to be 100% compatible to TeX. – Martin Schröder Feb 13 '16 at 23:37
• @AFeldman We are not talking about a run of xelatex producing the same file as a run of pdflatex!!! We are talking about multiple runs of xelatex on the same files, that is: running "xelatex file.tex" repeated time! – Paulo Ney Feb 14 '16 at 15:25

The issue is related to the driver: xdvipdfmx. In order to generate unique tags for fonts, random number is used. Try

xelatex -no-pdf test

xdvipdfmx test.xdv
pdffonts test.pdf


The tag will change like

LYKESP+CMR10
CBIVMK+CMR10
...


each time you run

xdvipdfmx test.xdv

• But this doesn't explain the difference in size, does it? – Robert Feb 20 '16 at 18:48
• @Robert: The data are compressed by zlib as a stream. Different data may be compressed differently. – Akira Kakuto Feb 20 '16 at 22:50
• @AkiraKakuto Indeed this is the core of the problem, but I still do not understand why "xdvipdfmx" needs to generate a "unique" tag per font! Aren't these fonts uniquely identified by their names? – Paulo Ney Feb 21 '16 at 21:43
• It’s a very old practice to prepend tags to subsetted fonts embedded in PostScript and PDF files. I think it was a recommendation early on (probably because the embedded font, being a subset, is formally not the same as the full font). – Arthur Reutenauer Feb 22 '16 at 13:27
• @PauloNey: In TeX Live 2016, you will obtain exactly the same pdf file from a unique xdv file, if you define an environment variable SOURCE_DATE_EPOCH correctly like SOURCE_DATE_EPOCH=1456304492. The name of an output pdf file must be the same. – Akira Kakuto Feb 24 '16 at 11:54

What seems to differ is the actual binary encodings of some parts. I trust that the pdf rendering does not vary at all. In my experience the file size changed only by plus or minus 1 byte (mac os x). I have done an ediff in hexadecimal mode of two such pdf's, here is a snapshot of where the first differences appear:

This first difference thus occurs in a part of the pdf relative to the partially embedded font. I don't know what causes this.

Generally speaking I can imagine that if you have say some 213 bytes thing which must be stored in 256 bytes, then the last 43 bytes may be random memory, if moreover one or many such things together are compressed then you get varying result. On undecompressing, there will be some varying random junk after structures terminators (or after a given number of bytes). Like non-coding DNA. Which perhaps is not so much non-coding, but let's not digress.

Only someone familiar with XeTeX source code can answer convincingly I guess.

Don't worry about the C-M-' undefined, I was trying to capture the thing via keyboard shortcuts I had forgotten.

• I have to admit that you flying over my head right now and I have not fully understood the meaning of everything you said ... nonetheless even if it is the compression if a font -- why would it compress differently on different runs? I still would expect the whole process to be deterministic and compress in exactly the same way. – Paulo Ney Feb 14 '16 at 18:09