latex pgfmath precision

What's the correct way to use pgfmath function? See below simple example:

\documentclass[border=1pt]{standalone}
\usepackage{tikz}
\begin{document}
\pgfmathsetmacro{\x}{3}
\pgfmathsetmacro{\y}{10}
\pgfmathsetlengthmacro{\z}{\x/\y}
$\x \div \y = \z$
\end{document}

The output is: 1. Obviously, expected outptu is 0.3.
2. How to remove the unit "pt"?
• On the second question, you can use \strip@pt. The first one I don't know the exact reason, sorry (but I have a feeling that "truncation" will be among the used words for the answer :) – Phelype Oleinik Apr 30 '18 at 15:03
• Why are you using pgfmathsetlength? – percusse Apr 30 '18 at 16:06

This is a common problem when calculating with non-integers on a computer. Simplified, they have a resolution, i.e. a smallest number they can handle, and all numbers are a multiple of this. This leads to all sorts of rounding errors, which can add up if extensive calculations are done.

PGF normally uses TeXs fixed point arithmetic, which uses 16bit for the fractional part. For computations the numbers are treated as integers. One of the issues with this is that divisions are the usual truncated integer divisions.

TeXs unit for the smallest possible number is a scaled point (sp), which is (1/65536)pt or 0.000015258789pt. 3pt is internally represented as 3*65536sp = 196608sp. Dividing this by 10 would be 19660.8sp, but the 0.8 is truncated leading to 19660sp. Scaling this to pt results in 0.299987792969pt, shown as '0.29999pt' since TeX rounds to 5 digits. If you enter 0.3 directly, this will be internally represented as 19661sp, which is 0.300003051758pt, shown as '0.3pt'.

If you need higher precision, you can take a look at PGFs fpu library.

Removing pt and rounding the output:

You used \pgfmathsetlengthmacro, which adds the unit pt to the number. Instead you should use \pgfmathsetmacro, which just stores the number (without the unit).

For rounding the number you can use PGFs number printing macros. Here \pgfmathprintnumberto is useful. In your example it will set the macro to '0.3'. You may want to take a look at the options /pgf/number format/fixed, /pgf/number format/fixed zerofill and /pgf/number format/precision for setting up the number printing.

Example code:

\documentclass[border=1pt]{standalone}
\usepackage{tikz}
\begin{document}
\pgfmathsetmacro{\x}{3}
\pgfmathsetmacro{\y}{10}
\begin{tabular}{ll}
with \verb|\pgfmathsetlengthmacro|
&
\pgfmathsetlengthmacro{\z}{\x/\y}
$\x \div \y = \z$
\\
with \verb|\pgfmathsethmacro|
&
\pgfmathsetmacro{\z}{\x/\y}
$\x \div \y = \z$
\\
with \verb|\pgfmathparese| and \verb|\pgfmathprintnumberto|
&
\pgfmathparse{\x/\y}\pgfmathprintnumberto{\pgfmathresult}{\z}
$\x \div \y = \z$
\\
with \verb|\pgfmathsethmacro| and \verb|\z| set to 0.3
&
\pgfmathsetmacro{\z}{0.3}
$\x \div \y = \z$
\end{tabular}
\end{document}

Result: • related: at tex.stackexchange.com/a/338510/4686 and tex.stackexchange.com/a/231281/4686 I provided some detailed explanations related to how TeX handles dimensions. There are some surprising twists: 0.22222pt is not the same internally as 0.222222pt despite the former being best decimal approximation to 5 digits, and \the on a dimension never prints more than 5 digits after decimal mark. (0.22223pt gives same as 0.222222pt despite looking more distant from it than 0.22222pt, but 0.33333pt does give same as 0.333333pt and not same as 0.33334pt) – user4686 May 1 '18 at 9:04