latex pgfmath precision

Question

What's the correct way to use pgfmath function? See below simple example:

\documentclass[border=1pt]{standalone}
\usepackage{tikz}
\begin{document}
\pgfmathsetmacro{\x}{3}
\pgfmathsetmacro{\y}{10}
\pgfmathsetlengthmacro{\z}{\x/\y}
$\x \div \y = \z$
\end{document}

The output is:

1. Obviously, expected outptu is 0.3.
2. How to remove the unit "pt"?

Answer 1

About precision:

This is a common problem when calculating with non-integers on a computer. Simplified, they have a resolution, i.e. a smallest number they can handle, and all numbers are a multiple of this. This leads to all sorts of rounding errors, which can add up if extensive calculations are done.

PGF normally uses TeXs fixed point arithmetic, which uses 16bit for the fractional part. For computations the numbers are treated as integers. One of the issues with this is that divisions are the usual truncated integer divisions.

TeXs unit for the smallest possible number is a scaled point (sp), which is (1/65536)pt or 0.000015258789pt. 3pt is internally represented as 3*65536sp = 196608sp. Dividing this by 10 would be 19660.8sp, but the 0.8 is truncated leading to 19660sp. Scaling this to pt results in 0.299987792969pt, shown as '0.29999pt' since TeX rounds to 5 digits. If you enter 0.3 directly, this will be internally represented as 19661sp, which is 0.300003051758pt, shown as '0.3pt'.

If you need higher precision, you can take a look at PGFs fpu library.

Removing pt and rounding the output:

You used \pgfmathsetlengthmacro, which adds the unit pt to the number. Instead you should use \pgfmathsetmacro, which just stores the number (without the unit).

For rounding the number you can use PGFs number printing macros. Here \pgfmathprintnumberto is useful. In your example it will set the macro to '0.3'. You may want to take a look at the options /pgf/number format/fixed, /pgf/number format/fixed zerofill and /pgf/number format/precision for setting up the number printing.

Example code:

\documentclass[border=1pt]{standalone}
\usepackage{tikz}
\begin{document}
\pgfmathsetmacro{\x}{3}
\pgfmathsetmacro{\y}{10}
\begin{tabular}{ll}
with \verb|\pgfmathsetlengthmacro|
&
\pgfmathsetlengthmacro{\z}{\x/\y}
$\x \div \y = \z$
\\
with \verb|\pgfmathsethmacro|
&
\pgfmathsetmacro{\z}{\x/\y}
$\x \div \y = \z$
\\
with \verb|\pgfmathparese| and \verb|\pgfmathprintnumberto|
&
\pgfmathparse{\x/\y}\pgfmathprintnumberto{\pgfmathresult}{\z}
$\x \div \y = \z$
\\
with \verb|\pgfmathsethmacro| and \verb|\z| set to 0.3
&
\pgfmathsetmacro{\z}{0.3}
$\x \div \y = \z$
\end{tabular}
\end{document}

Result: