pgfplotstable, round-off error

Question

I'd like to do some simple computation in my table and it looks like I'm seeing some round-off errors.

Does anybody know the precision that pgfplotstable uses internally? Is there any work around besides that following really ugly hack, that I actually don't know, why it's working at all?

Here's my concrete MWE:

\documentclass{article}
\usepackage{pgfplotstable}
\begin{document}
  \pgfplotstableread[col sep = comma]{
    a,         b,
    640000,    3200,  
    1920000,   3200,
    3200000,   3200,
    6400000,   3200,
    12800000,  3200,
  } \data
  \begin{table}
    \pgfplotstableset{create on use/c/.style={create col/expr={2*\thisrow{a}+5*\thisrow{b}}}}
   \pgfplotstableset{create on use/d/.style={create col/expr={(2*(\thisrow{a}*256)+5*(\thisrow{b}*256))/256}}}
   \pgfplotstabletypeset[columns={a,b,c,d},
     columns/a/.style={column name={a}, fixed, precision=0, column type={r|}, },
     columns/b/.style={column name={b}, fixed, precision=0, column type={r|}, },
     columns/c/.style={column name={formular}, fixed, precision=0, column type={r|}, },
     columns/d/.style={column name={Hack}, fixed, precision=0, column type={r|}, },
   ]{\data}
 \end{table}
\end{document}

Thanks a lot, Juhui

Answer 1

This is not really an answer, but a workaround that might be feasible for you: If you can use lualatex instead of pdflatex, you can use a \directlua statement to do the calculations:

\documentclass{article}
\usepackage{pgfplotstable}
\begin{document}
  \pgfplotstableread[col sep = comma]{
    a,         b,
    640000,    3200,  
    1920000,   3200,
    3200000,   3200,
    6400000,   3200,
    12800000,  3200,
  } \data


  \begin{table}
    \pgfplotstableset{create on use/c/.style={create col/expr={\directlua{tex.sprint(2*\thisrow{a}+5*\thisrow{b})}}}}
   \pgfplotstableset{create on use/d/.style={create col/expr={(2*(\thisrow{a}*256)+5*(\thisrow{b}*256))/256}}}
   \pgfplotstabletypeset[columns={a,b,c,d},
     columns/a/.style={column name={a}, fixed, precision=0, column type={r|}, },
     columns/b/.style={column name={b}, fixed, precision=0, column type={r|}, },
     columns/c/.style={column name={Lua}, fixed, precision=0, column type={r|}, },
     columns/d/.style={column name={Hack}, fixed, precision=0, column type={r|}, },
   ]{\data}
 \end{table}
\end{document}

Answer 2

This is a (standard) rounding issue. You would probably see the same if you would compute these numbers in a C program with floats.

A solution is either to use a different math parser which has higher precision (like Jake's answer) or to round the result.

Since you cannot easily increase the precision of the routines in pgfplotstable, you can round the result... which is feasible for this case. The solution would be to use fixed relative: it rounds to the first <precision> nonzero digits. If you use 5 or 6 as precision, you have the precision of the math parser. That is quite reasonably IMHO.

\documentclass{article}
\usepackage{pgfplotstable}
\begin{document}
  \pgfplotstableread[col sep = comma]{
    a,         b,
    640000,    3200,  
    1920000,   3200,
    3200000,   3200,
    6400000,   3200,
    12800000,  3200,
  } \data
  \begin{table}
    \pgfplotstableset{create on use/c/.style={create col/expr={2*\thisrow{a}+5*\thisrow{b}}}}
   \pgfplotstableset{create on use/d/.style={create col/expr={(2*(\thisrow{a}*256)+5*(\thisrow{b}*256))/256}}}
   \pgfplotstabletypeset[columns={a,b,c,c,d},
     columns/a/.style={column name={a}, fixed, precision=0, column type={r|}, },
     columns/b/.style={column name={b}, fixed, precision=0, column type={r|}, },
     columns/c/.style={column name={formular}, fixed, precision=0, column type={r|}, },
     display columns/3/.style={
        column name={formular*}, 
        fixed relative, precision=5, 
        column type={r|},
     },
     columns/d/.style={column name={Hack}, fixed, precision=0, column type={r|}, },
   ]{\data}
 \end{table}
\end{document}

Note that there is also a relative style in the number printer. That would be even better here -- but I designed it in a stupid way and a fix will only be available in the next version (currently, it is useful for numbers less than 1).