How to prevent loss of precision for operations on length values?

Value \MM is equal to 1mm and is internally stored as 2.84526pt. Value \MMQUARTER is equal to 0.25mm and is supposed to be stored as 0.711315pt (2.84526/4 = 0.711315) but is stored as 0.7113pt. I need to find a solution to prevent the truncation of 0.000015.

\documentclass[border=5mm]{standalone}
\usepackage{tikz}

\begin{document}
\begin{tikzpicture}

\pgfmathsetlengthmacro{\MM}{1mm}
\pgfmathsetlengthmacro{\MMQUARTER}{0.25mm}

\path node[align=left]
{%
\pgfmathparse
{% 'MM' IS 2.84526
\MM
}   \pgfmathresult
% NEXT LINE CAN BE IGNORED
\hspace{0.33cm} = 1mm
\\ % NEW LINE

\pgfmathparse
{% 'MMQUARTER' MUST BE 0.711315 BUT IS 0.7113
\MMQUARTER
}   \pgfmathresult
% NEXT 3 LINES CAN BE IGNORED
\hspace{0.5cm} $\approx$ 0.25mm;
\\ % NEW LINE
\hspace{1.5cm} must be 0.7113\textcolor{red}{15} (exactly 0.25mm)
};

\end{tikzpicture}
\end{document} • Actually, I don't think there's much you can do about it. If you need 0.711315pt, better set 0.711315pt and not 0.25mm. This has nothing to do with PGF maths, I don't think. It is just a limitation of TeX. \newdimen\mydim \mydim=0.25mm also sets a dimension register to 0.7113pt. And PGF uses the standard means for dimensions, even if you load fixedpointarithmetic, while anything you do in fpu has to be translated back into the regular stuff for PGF, so I think you'd have the same issue there, though I'm not sure. – cfr Sep 12 '17 at 3:56
• If you set 0.711315pt you'll get 0.71132pt, because it is rounding to 5dp anyway. – cfr Sep 12 '17 at 3:58
• See tex.stackexchange.com/questions/135534/…, including the links in the question (which may be more helpful than this particular Q&A itself). Lua might be an option: tex.stackexchange.com/questions/196171/… or, possibly, use pgfplots to replace \pgfmathparse with a Lua version, tex.stackexchange.com/questions/333633/…. – cfr Sep 12 '17 at 4:04
• However, I only get 0.711304pt with Lua. But that's probably due to use of \pgfmathsetlength rather than \pgfmathsetmacro. – cfr Sep 12 '17 at 4:13

The first sentence is a bit misleading

Value \MM is equal to 1mm and is internally stored as 2.84526pt

TeX (almost) never uses floating point arithmetic, and never uses it at a place where any difference in machine arithmetic would be detectable from within TeX.

Almost all arithemtic is exact integer arithmetic with decimal conversion just for display purposes.

TeX uses the conversion 25.4mm=1in=72.27pt, 65536sp=1pt

so 1mm is 186467sp and it is this integer that is stored for the length

dividing by 4 and rounding down gets 46616sp which is displayed as 0.7113pt.

You can see the values interactively by running tex on the commandline:

$tex This is TeX, Version 3.14159265 (TeX Live 2017) (preloaded format=tex) **\dimen0=1mm *\count0=\dimen0 *\showthe\count0 > 186467. <*> \showthe\count0 ? *\divide\count0 by 4 *\showthe\count0 > 46616. <*> \showthe\count0 ? *\dimen0=\count0 sp *\showthe\dimen0 > 0.7113pt. <*> \showthe\dimen0 ? x No pages of output. Transcript written on texput.log. • TeX is truncating, rather than rounding, IIRC. – egreg Sep 12 '17 at 10:21 • @egreg well yes that's what I meant by rounding down, terminology being a bit variable here (it rounds towards zero rather than down, really) which is also known as truncating. – David Carlisle Sep 12 '17 at 12:08 • I think the answer is a bit misleading as it suggest that 0.25mm is handled as one- quarter of a mm. Although this does work with other units, it appears a bit coincidental. I found the example of 0.0125in which is not the same as division of 1in by 80 (truncating): 59189sp for \dimexpr0.0125in versus 59203sp for dividing 1in by 80. Even simpler: try 0.1in and compare with dividing 1in by 10 (473657sp vs 473628sp) – user4686 Sep 12 '17 at 14:08 • @jfbu yes I may reword a bit, mostly I just wanted to point out that tex is using integer arithmetic not floating point. It's true that .25\somelength isn't always same as \divide\somelength by 4 (and I showed one when the question had the other...) – David Carlisle Sep 12 '17 at 15:04 • yes I took your answer as starting point as you had already pointed out the most important which is that internally, integers are used, not floating point operations. – user4686 Sep 12 '17 at 15:08 Let me first comment on the comments in your code \pgfmathsetlengthmacro{\MM}{1mm} \pgfmathsetlengthmacro{\MMQUARTER}{0.25mm} {% 'MM' IS 2.84526 \MM {% 'MMQUARTER' MUST BE 0.711315 BUT IS 0.7113 \MMQUARTER I assume that \pgfmathsetlengthmacro boils down internally at some point to standard TeX. A dimension specification like 0.25mm is parsed this way. I will describe it in terms not corresponding one-by-one to the TeX processing but producing exactly equivalent result. 1. first 0.25 is rounded to an integral multiple N/65536. Here this is an exact match N=16384. 2. then a conversion factor is applied, which in the case of the mm unit is (according to tex.web §458) 7227/2540. This means that TeX evaluates the truncation to an integer of the exact fraction 16384*7227/2540. Now:$ rlwrap etex
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/MacPorts 2017_1) (preloaded format=etex)
restricted \write18 enabled.
**xintexpr.sty
entering extended mode
(/usr/local/texlive/2017/texmf-dist/tex/generic/xint/xintexpr.sty
(/usr/local/texlive/2017/texmf-dist/tex/generic/xint/xintfrac.sty
(/usr/local/texlive/2017/texmf-dist/tex/generic/xint/xint.sty
(/usr/local/texlive/2017/texmf-dist/tex/generic/xint/xintcore.sty
(/usr/local/texlive/2017/texmf-dist/tex/generic/xint/xintkernel.sty))))
(/usr/local/texlive/2017/texmf-dist/tex/generic/xint/xinttools.sty))
*\message{\xinttheiexpr  16384*7227/2540\relax}
46616.9952755905511811023622047244094488188976
*\bye
No pages of output.
Transcript written on xintexpr.log.

So in this case TeX will obtain 46616 and we see that it makes here a relatively big error.

3. The conclusion is that 0.25mm is stored internally as 46616sp where 1sp is the 65536th part of a (TeX) pt.

For more details see Why pdf file cannot be reproduced?

Then, what happens when a dimension internally stored as 46616sp is displayed using \the ? The algorithm is in §103 of tex.web and is a bit subtle. We want to produce a value in pt unit which would re-produce exactly the 46616sp if rounded to an integral multiple of 1/65536. Of course first we handle the integer part, which here vanishes. So let's assume we have some initial n with 0 <= n < 65536. Knuth actually considers (indirectly) the decimal expansion of (n + 0.5)/65536. Let's say we look at the first K digits of this decimal expansion, so, for K = 1, 2, 3, 4 and at most 5, we have

(n+0.5)/65536 = D_K/10^K + N_K/(10^K \times 65536)

where actually N_K for those K considered is an integer, positive, and strictly less than 65536 (power of two considerations justify some of my assertions). The N_K are simply obtained by repeated multiplication by 10 modulo 65536. The equation above gives

65536 D_K/10^K = n + 0.5 - N_K/10^K

and if for any K we observe that 0< N_K <= 10^K (positivity is known), the rounding of 65536 D_K/10^K will give back the n as expected. This might happen for K = 1, ... or 4. Imagine it did not. Then it will necessarily at K = 5 but this case is handled especially. If we reach K = 5 it is because we had N_4 > 10000. Then let's M_5 = 10 N_4 - 50000 + 32768. It is > 65536, so we write M_5 = q 65536 + r where q is a digit. After simplification we obtain

n = 65536 (10 D_4 + q)/100000 + (r - 32768)/100000

and with D_5 = 10 D_4 + q which has a fifth digit, we again get a decimal D_5/100000 whose rounding as integral multiple of 1/65536 produces n/65536. (I will have to read again §103 of tex.web but this is my impression of what it boils down to).

We can follow the steps for n = 46616. First we compute 10(46616.5) = 466165 = 7x65536+7413 which gives use N_1 = 7413. It is not <= 10, so algorithm continues. First digit was 7.

Python 3.5.4 |Continuum Analytics, Inc.| (default, Aug 14 2017, 12:43:10)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
>>> divmod(74130,65536)
(1, 8594)
>>> divmod(85940,65536)
(1, 20404)
>>> divmod(204040,65536)
(3, 7432)

thus digits are 7, 1, 1, 3, and here we reach N_4 = 7432 which is less than 10000. Algorithm stops here and TeX thus produces 0.7113pt.

From the description above you see it will never produce 6 digits after decimal mark such as 0.711315. To obtain this you must use a math engine (possibly implemented in macros; since e-TeX of 1999 with \numexpr it is easier to do this expandably, at least theoretically) able to handle decimal numbers like those.

You can use \xinttheiexpr [D] macro for such computations where D is the asked-for number of digits after decimal mark of final result, but computations are done exactly.

Going back to the quarter of millimeter (using Plain TeX, works in LaTeX too with \usepackage{xintexpr}):

\input xintexpr.sty

\xintdefvar mm:= 7227/2540;

\edef\MYQUARTERMM{\xinttheexpr trunc(0.25mm, 40)\relax}

\show\MYQUARTERMM

\bye

produces in the log

> \MYQUARTERMM=macro:
->0.7113188976377952755905511811023622047244.
l.7 \show\MYQUARTERMM

This shows by the way that the meaning of 0.711315 as arising in the original question isn't that clear.

From the above, an a priori closer 6 digits approximation is 0.711319pt. After checking one sees though that both 0.711319pt and 0.711315pt get converted by TeX into the internal representation 46617sp (\number\dimexpr 0.711319pt\relax, \number\dimexpr 0.711315pt\relax).

And \the\dimexpr 0.711319pt\relax outputs 0.71132pt. Thus 0.71132pt is the most accurate 5 digits representation (after testing it is the only one giving 46617sp internally) of the actual exact 0.25mm dimension. However as explained above, when TeX encounters 0.25mm it will obtain 46616sp. Using the bigger cm unit as in 0.025cm gives a greater loss of precision (46605sp).

• technical note: "rounding" means "with ties going away from zero". In other contexts "rounding" in numerical algorithms may mean "with ties going to even". But here 10.5 rounded means 11 not 10. – user4686 Sep 12 '17 at 12:18
• rant: division in \numexpr "rounds" in the sense of my previous comment; but TeX division (\divide) truncates. In nine cases of out of ten, the rounding done by / in \numexpr has proved a nuisance in my experience, but well. – user4686 Sep 12 '17 at 12:19
• One can prove that Knuth algorithms, if starting with 0.x pt, 0.xy pt, 0.xyz pt or 0.xyzt pt where the last digit is non-zero, going to a dimen register and back to <decimal>pt using \the always reproduce exactly the initial specification. This is wrong starting with 5 digits. Of course with 6 digits or more, we know this is wrong because the procedure gives in the end at most 5 digits after decimal mark. In the case of 5 digits, examples such as 0.63001pt or 0.28999pt show that the input is not necessarily reproduced. – user4686 Sep 12 '17 at 13:57