\pgfplotstabletypeset heatmap: logarithmic scale

Question

I am trying to create my own rendition of this heatmap. My values range anywhere from 1 to 783764. I want to plot them as colors on a logarithmically (base10) scaled colormap. So the values' logarithmic representations should range from 0 to ~5.894.

When I set the max value color cells={min=1,max=783764}, I get the following error

! Dimension too large.
<to be read again>
\relax
l.104 ]{data.csv}

Through experimentation I have fond that 16383 appears to be the maximum value that does not give me an error. Since I don't know how to make my intended maximum of 783764 work, I attempted to do some preprocessing (see pgfplotstable docs Sec. 3.3), and pull all my values through my own lg10 function: preproc/expr={lg10(##1)}
However, that then gave me another error:

! Package PGF Math Error: Sorry, an internal routine of the floating point unit got an ill-formatted floating point number `1'. The unreadable part was near '1'..

First of all, this is weird, since there is no value 1 in the data table that I tried this on. Secondly, I tried appending .0 to all my (integer) values, but that made no difference.

How can I make this work?

Here's my Minimum non-working example:

\documentclass{standalone}

\listfiles
\usepackage{colortbl}
\usepackage{filecontents}
\usepackage{pgfplotstable}
\usetikzlibrary{pgfplots.colormaps}

\pgfplotsset{compat=1.9}

% https://tex.stackexchange.com/a/83865/31877
\pgfplotstableset{
    /color cells/min/.initial=0,
    /color cells/max/.initial=1000,
    /color cells/textcolor/.initial=,
    %
    % Usage: 'color cells={min=<value which is mapped to lowest color>, 
    %   max = <value which is mapped to largest>}
    color cells/.code={%
        \pgfqkeys{/color cells}{#1}%
        \pgfkeysalso{%
            postproc cell content/.code={%
                \begingroup
                % acquire the value before any number printer changed it:
                \pgfkeysgetvalue{/pgfplots/table/@preprocessed cell content}\value
                \ifx\value\empty
                    \endgroup
                \else
                \pgfmathfloatparsenumber{\value}%
                \pgfmathfloattofixed{\pgfmathresult}%
                \let\value=\pgfmathresult
                % map that value:
                \pgfplotscolormapaccess
                    [\pgfkeysvalueof{/color cells/min}:\pgfkeysvalueof{/color cells/max}]
                    {\value}
                    {\pgfkeysvalueof{/pgfplots/colormap name}}%
                % now, \pgfmathresult contains {<R>,<G>,<B>}
                % 
                % acquire the value AFTER any preprocessor or
                % typesetter (like number printer) worked on it:
                \pgfkeysgetvalue{/pgfplots/table/@cell content}\typesetvalue
                \pgfkeysgetvalue{/color cells/textcolor}\textcolorvalue
                % tex-expansion control
                \toks0=\expandafter{\typesetvalue}%
                \xdef\temp{%
                    \noexpand\pgfkeysalso{%
                        @cell content={%
                            \noexpand\cellcolor[rgb]{\pgfmathresult}%
                            \noexpand\definecolor{mapped color}{rgb}{\pgfmathresult}%
                            \ifx\textcolorvalue\empty
                            \else
                                \noexpand\color{\textcolorvalue}%
                            \fi
                            \the\toks0 %
                        }%
                    }%
                }%
                \endgroup
                \temp
                \fi
            }%
        }%
    }
}

\pgfmathdeclarefunction{lg10}{1}{%
    \pgfmathparse{ln(#1)/ln(10)}%
}

\begin{filecontents}{data.csv}
item,Xa,Xb,Ya,Yb,Za,Zc
a,2,,2,,,
b,,2,,,4,2
c,,5,,,,
d,2,,8,,,
e,,,,,5,5
f,,,,3,,
g,,2,,,,
h,6,,,4,,
i,,,4,,,
j,,,,,,2
k,,2,,,3,2
l,,,7,,,
m,,6,,3,,
n,,,,,,7
\end{filecontents}

\begin{document}


\pgfplotstabletypeset[
    %font=\tiny,
    colorCell/.style={
        preproc/expr={lg10(##1)},%fails to work on ANY number
        color cells={min=1,max=16383}%max should be 783764
    },
    itemCell/.style={
        string type,
        column name={},
    },
    col sep=comma,
    columns={item,Xa,Xb,Ya,Yb,Za,Zc},
    columns/item/.style={itemCell},
    columns/Xa/.style={colorCell},
    columns/Xb/.style={colorCell},
    columns/Ya/.style={colorCell},
    columns/Yb/.style={colorCell},
    columns/Za/.style={colorCell},
    columns/Zb/.style={colorCell},
]{data.csv}

\end{document}

Answer 1

The problem is in the style posted at https://tex.stackexchange.com/a/83865/31877 : apparently, the color map access method expects a number which is not in float representation. A quickfix would be to write \pgfkeys{/pgf/fpu=false} into the color cells style.

But: I suppose you want to typeset the raw number, not its log, right? You could also apply the log inside of the color cells style. That's what I did here - it does not need a special preprocessing step:

\documentclass{standalone}

\listfiles
\usepackage{colortbl}
\usepackage{filecontents}
\usepackage{pgfplotstable}
\usetikzlibrary{pgfplots.colormaps}

\pgfplotsset{compat=1.9}

% https://tex.stackexchange.com/a/83865/31877
\pgfplotstableset{
    /color cells/min/.initial=0,
    /color cells/max/.initial=1000,
    /color cells/textcolor/.initial=,
    %
    % Usage: 'color cells={min=<value which is mapped to lowest color>, 
    %   max = <value which is mapped to largest>}
    color cells/.code={%
        \pgfqkeys{/color cells}{#1}%
        \pgfkeysalso{%
            postproc cell content/.code={%
                \begingroup
                % acquire the value before any number printer changed it:
                \pgfkeysgetvalue{/pgfplots/table/@preprocessed cell content}\value
                \ifx\value\empty
                    \endgroup
                \else
                \pgfkeys{/pgf/fpu=true}%
                %\pgfmathfloatparsenumber{\value}%
                \pgfmathparse{log10(\value)}%
                \pgfmathfloattofixed{\pgfmathresult}%
                \let\value=\pgfmathresult
                \pgfkeys{/pgf/fpu=false}% this here was the problem
                % map that value:
                \pgfplotscolormapaccess
                    [\pgfkeysvalueof{/color cells/min}:\pgfkeysvalueof{/color cells/max}]
                    {\value}
                    {\pgfkeysvalueof{/pgfplots/colormap name}}%
                % now, \pgfmathresult contains {<R>,<G>,<B>}
                % 
                % acquire the value AFTER any preprocessor or
                % typesetter (like number printer) worked on it:
                \pgfkeysgetvalue{/pgfplots/table/@cell content}\typesetvalue
                \pgfkeysgetvalue{/color cells/textcolor}\textcolorvalue
                % tex-expansion control
                \toks0=\expandafter{\typesetvalue}%
                \xdef\temp{%
                    \noexpand\pgfkeysalso{%
                        @cell content={%
                            \noexpand\cellcolor[rgb]{\pgfmathresult}%
                            \noexpand\definecolor{mapped color}{rgb}{\pgfmathresult}%
                            \ifx\textcolorvalue\empty
                            \else
                                \noexpand\color{\textcolorvalue}%
                            \fi
                            \the\toks0 %
                        }%
                    }%
                }%
                \endgroup
                \temp
                \fi
            }%
        }%
    }
}

\pgfmathdeclarefunction{lg10}{1}{%
    \pgfmathparse{ln(#1)/ln(10)}%
}

\begin{filecontents}{data.csv}
item,Xa,Xb,Ya,Yb,Za,Zc
a,2,,2,,,
b,,2,,,4,2
c,,5,,,,
d,2,,8,,,
e,,,,,5,5
f,,,,3,,
g,,2,,,,
h,6,,,4,,
i,,,4,,,
j,,,,,,2
k,,2,,,3,2
l,,,7,,,
m,,6,,3,,
n,,,,,,7
\end{filecontents}

\begin{document}


\pgfplotstabletypeset[
    %font=\tiny,
    colorCell/.style={
      %  preproc/expr={lg10(##1)},%fails to work on ANY number
        color cells={min=0.3,max=0.85}%max should be 783764
    },
    itemCell/.style={
        string type,
        column name={},
    },
    col sep=comma,
    columns={item,Xa,Xb,Ya,Yb,Za,Zc},
    columns/item/.style={itemCell},
    columns/Xa/.style={colorCell},
    columns/Xb/.style={colorCell},
    columns/Ya/.style={colorCell},
    columns/Yb/.style={colorCell},
    columns/Za/.style={colorCell},
    columns/Zc/.style={colorCell},
]{data.csv}

\end{document}