Easiest way to plot matrix image

Question

Is there an easier way to create this in pgfplots than plotting the matrix image externally and using addplot graphics (as on p.42 of the pgfplots manual)? A mesh plot (as suggested for plotting matrixes here) might not support the number of data points I'd like to use, plotting 1000x500 points this way won't work, or create a giantic output file.

Getting the upper/lower limits for the colorbar and the palette right isn't that straightforward when using two plotting programs (the gnuplot print statement can be redirected into a file, which allows generating TeX-code from gnuplot, so the limits as determined by gnuplot could be imported as macros etc.)

Since pgfplots can call gnuplot for line-data already, passing commands and reading back the output, could I use this to write the gnuplot code to generate the image directly into my TeX file instead of having to run an external script and manage the temporary files manually?

set terminal tikz standalone externalimages
set output "plot.tex"
plot "data.dat" matrix with image
set print "plot-params.tex"
print sprintf("\cb{%g}{%g}",GPVAL_CB_MIN,GPVAL_CB_MAX)

The style of gnuplot's tikz output doesn't match pgfplots' style and draws the color bars as a series of boxes instead of a gradient but it automatically generates the pixel data as plot.0.png in the correct size, which makes it a bit easier to use with addplot graphics.

This gets worse when multiple such plots should be combined using multiplot/groupplots.

Answer 1

• As the sample matrix datasets(131*131) are large in size LuaLaTeX -shell-escape approach was adopted and pdflatex fails due to memory limitations. I have retained the TeX memory defaults on TeXLive 2012 frozen. In view of your real datasets of size (1000x500) points. I made some points to help you.

• TeX memory problems in pgfplots can be resolved by(More info at Section 6.1 pgfplots manual revision 1.8, 17/3/2013)

  1)How to expand TeX's "main memory size"? (pgfplots memory overload) and How can one predict pgfplots memory overload?

  2)Downsampling the data using gnuplot or sed as discussed in pgfplot: plotting a large dataset.

  3)Export separate figure pdf's using pgfplots externalize library to reduce the compilation time and avoid memory problems for large documents.

  4)Alternatively use fully gnuplot solution using gnuplottex package via improved terminal exports epslatex, pdf, and lua/tikz.

gnuplot lua/tikz terminal driver

"The lua terminal driver creates data intended to be further processed by a script in the lua programming language. At this point only one such lua script, gnuplot-tikz.lua, is available. It produces a TeX document suitable for use with the latex TikZ package. Other lua scripts could be written to process the gnuplot output for use with other TeX packages, or with other non-TeX tools. set term tikz is shorthand for set term lua tikz. As decribed above, it uses the generic lua terminal and an external lua script to produce a latex document" from gnuplot 4.6 documentation

Examples:

A tikz terminal can export full-plot.tex of plot like in this TeXample.net example and gnuplottikz example

Hoping this works for you.

Using pgfplots (invoking gnuplot) and it's groupplot library. on 131*131 matrix dataset provided by OP

\documentclass[preview=true,12pt,border=2pt]{standalone}
\usepackage{pgfplots}
\pgfplotsset{compat=newest}
\usepgfplotslibrary{groupplots} 
%\usepgfplotslibrary{external}
%\tikzexternalize% activate externalization!
% same matrix data used for all four figures for illustration
% code compiled with lualatex -shell-escape, gnuplot 4.4.3 and pgfplots 1.8 
% Matrix dataset(131*131) in "mat-data.dat" (provided by OP in comment). 
\begin{document}
\begin{tikzpicture}
\begin{groupplot}[view={0}{90},colorbar,colormap/bluered,colorbar style={%
ytick={-19,-18,-17,-16,-15,-14,-13,-12,-11}},label style={font=\small},group style={group size=2 by 2,horizontal sep=3cm},
height=3.5cm,width=3.5cm,footnotesize]
\nextgroupplot % First Figure
\addplot3[raw gnuplot,surf,shader=interp]
            gnuplot[id={surf}]{%
set pm3d map interpolate 0,0;
splot 'mat-data.dat' matrix using 1:2:(log($3));}; %natural logarithm
\nextgroupplot % Second Figure
\addplot3[raw gnuplot,surf,shader=interp]
            gnuplot[id={surf}]{%
set pm3d map interpolate 0,0;
splot 'mat-data.dat' matrix using 1:2:(log($3));};
\nextgroupplot % Third Figure
\addplot3[raw gnuplot,surf,shader=interp]
            gnuplot[id={surf}]{%
set pm3d map interpolate 0,0;
splot 'mat-data.dat' matrix using 1:2:(log($3));};
\nextgroupplot % Fourth Figure
\addplot3[raw gnuplot,surf,shader=interp]
            gnuplot[id={surf}]{%
set pm3d map interpolate 0,0;
splot 'mat-data.dat' matrix using 1:2:(log($3));};
\end{groupplot}
\end{tikzpicture}
\end{document}

code compiled with lualatex -shell-escape, gnuplot 4.4.3 and pgfplots 1.8

Alternative Approach(option 4 above): Using fully gnuplot based on gnuplottex package

code compiled with pdflatex -shell-escape, gnuplot 4.4.3 and gnuplottex(version October 2, 2012)

\documentclass[preview=true,border=2pt]{standalone}
\usepackage{gnuplottex} 
\begin{document}    
\begin{gnuplot}[terminal=epslatex,terminaloptions={font ",7"}]
set tmargin 0       # set white top margin in multiplot figure
set bmargin 1       # set white bottom margin in multiplot figure  
set lmargin 0
set rmargin 0 
set xrange [0:130]; # x and y axis range 
set yrange [0:130];
set cbrange [-19:-12]; # colorbox range  
unset key;             # disables plot legend or  use "notitle"  
set palette model RGB defined (0 '#0000B4',1 '#00FFFF',2 '#64FF00',3 '#FFFF00',4 '#FF0000',5 '#800000');         # hex colormap matching pgfplots "colormap/bluered"  
set pm3d map;       # `splot` for drawing palette-mapped 3d colormaps/surfaces
set pm3d interpolate 0,0; # interpolate optimal number of grid points into finer mesh, and          
                          # color each quadrangle with 0,0
set multiplot layout 2,2 rowsfirst scale 0.9,1.2;  # subplots 2 by 2
# First Figure
splot 'mat-data.dat' matrix using 1:2:(log($3)); # matrix data with log(z-axis)
# Second Figure
splot 'mat-data.dat' matrix using 1:2:(log($3)); 
# Third Figure
splot 'mat-data.dat' matrix using 1:2:(log($3)); 
# Fourth Figure
splot 'mat-data.dat' matrix using 1:2:(log($3)); 
unset multiplot;
\end{gnuplot}
\end{document}

Alternative Approach: Reading from mat-data.dat file directly in pgfplots

I have failed to get result in this way. Anybody can give their feedback. \addplot3[surf,mesh/cols=131,mesh/ordering=rowwise,shader=interp] file {mat-data.dat};

Answer 2

Below is my current solution using an external script. It uses gnuplot to generate the raster data and prints the code that should be piped into a file and included into the document.

#!/bin/bash
cat <<EOF
  \begin{tikzpicture}
  \begin{groupplot}[
    group stype={group size=2 by 2},
    colorbar, colormap/bluered
  ]
EOF

function plot_matrix {
  matrix=${1/.dat/.png}
  gnuplot <<EOF
    set terminal tikz externalimages
    set output "temp.tex"
    unset xtics
    unset ytics # messes up xrange,yrange
    plot "$1" matrix w image
    # gnuplot uses <index> <r:0-1> <g> <b> syntax
    set palette defined (0 0 0 0.7, 1 0 1 1, 2 0.4 1 0, 3 1 1 0, 4 1 0 0, 5 0.5 0 0)
    set print '-' # default is stderr
    cb = sprintf('point meta min=%g,point meta max=%g',GPVAL_CB_MIN,GPVAL_CB_MAX)
    ra = sprintf('xmin=%g,xmax=%g,ymin=%g,ymax=%g',GPVAL_X_MIN,GPVAL_X_MAX,GPVAL_Y_MIN,GPVAL_Y_MAX)
    print sprintf('\nextgroupplot[%s,%s,$2]',cb,ra)
    print sprintf('\addplot graphics[%s]{$matrix};',ra)
EOF
  rm 'temp.tex'
  mv 'temp.01.png' $matrix
}

plot_matrix "a.dat" "ylabel={Y1}"
plot_matrix "b.dat" ""
plot_matrix "c.dat" "ylabel={Y2}"
plot_matrix "d.dat" ""

cat <<EOF
  \end{groupplot}
  \end{tikzpicture}
EOF

Answer 3

As another option, the Asymptote can handle bigger images. p.asy:

real unit=0.5mm;
unitsize(unit);
import graph;
import palette;

file fin=input("bigdata.dat");
real[][] v=fin.dimension(0,1310);
v=transpose(v);

int n=v.length;
int m=v[0].length;

write(n,m);

scale(Linear,Linear,Log);

pen[] Palette=
  Gradient(
    rgb(0,0,0.1)
    ,rgb(0,0,1)
    ,rgb(1,0,0)
    ,rgb(0,1,0)
    ,rgb(1,0.1,0)
    ,rgb(1,1,0)
  );

picture bar;
bar.unitsize(unit);

bounds range=image(v, (0,0),(n,m),Palette);

palette(bar,"$A$",range,(0,0),(50,n),Right,Palette,
  PaletteTicks(scale(10)*"$%0.1e$",pTick=black+5pt,Step=0.5e-6)
);

xaxis(0,m,RightTicks(scale(10)*"$%0f$",Step=200,step=100,beginlabel=false,black+5pt));
yaxis(0,n,LeftTicks(scale(10)*"$%0f$",Step=200,step=100,black+5pt));

add(bar.fit(),point(E),E);

The original 131x131 matrix data.dat was scaled to 1310x1310 by this helper pre.asy:

file fin=input("data.dat");
real[][] v=fin.dimension(0,131);
v=transpose(v);

int n=v.length;
int m=v[0].length;

real[][] w=new real[10n][10m];

file fout=output("bigdata.dat");
string s;

for(int i=0;i<10n;++i){
  s="";
  for(int j=0;j<10m;++j){
    w[i][j]=v[rand()%n][rand()%m]*0.618+v[i%n][j%n]*0.382;
    w[i][j]*=((real)(i+1)/10n+(real)(j+1)/10m)/2;
    s=format("%#.5e ",w[i][j]);
    write(fout,s);
  }
  write(fout,'\n');
}

Edit: There is nothing special in pre.asy, it was used just to get a bigger dataset bigdata.dat (1310x1310, about 20Mb), to check how asy can handle it. Btw, it the OP already has the 1000x500 file, it would be better to try it instead.

Comments on pre.asy:

file fin=input("data.dat");
real[][] v=fin.dimension(0,131);  // read data file into the matrix v[][]
v=transpose(v);                   

int n=v.length;                   // n - number of rows
int m=v[0].length;                // m - number of columns

This is a standard sequence to read a matrix from the data file. Now, declare a new matrix w, ten times bigger:

real[][] w=new real[10n][10m];

Declare the output file fout and string s:

file fout=output("bigdata.dat");
string s;

Next, there are C-like for loops to run through all indices of the new bigger matrix:

for(int i=0;i<10n;++i){
  s="";
  for(int j=0;j<10m;++j){

Now, put something into the current element w[i][j]:

    w[i][j]=v[rand()%n][rand()%m]*0.618+v[i%n][j%n]*0.382;
    w[i][j]*=((real)(i+1)/10n+(real)(j+1)/10m)/2;

some random element of the original small matrix is used, but it can be anything. In fact, it could be possible to calculate the entire matrix here, if the data used by OP are not coming from the sensors or use some really tricky algorithms.

    s=format("%#.5e ",w[i][j]); // format the value 
                                // according to scientific notation
                                // with 5 digits, e.g. `4.75586e-08`
    write(fout,s);              // write it to the file
  }
  write(fout,'\n');             // write a new line symbol
}

That's it.

And for the sake of comparison, this is the original 131x131 matrix, with (natural) log applied to the values, and a palette similar (I hope) to colormap/bluered in pgfplots:

real unit=0.5mm;
unitsize(unit);
import graph;
import palette;

file fin=input("data.dat");
real[][] v=fin.dimension(0,131);
v=transpose(v);

int n=v.length;
int m=v[0].length;

write(n,m);

for(int i=0;i<n;++i){
  for(int j=0;j<m;++j){
    v[i][j]=log(v[i][j]);
  }
}

//\pgfplotsset{
//colormap={bluered}{
//rgb255(0cm)=(0,0,180); rgb255(1cm)=(0,255,255); rgb255(2cm)=(100,255,0);
//rgb255(3cm)=(255,255,0); rgb255(4cm)=(255,0,0); rgb255(5cm)=(128,0,0)}
//}

pen[] Palette=
  Gradient(
    rgb(0,0,180.0/255)
    ,rgb(0,1,1)
    ,rgb(100.0/255,1,0)
    ,rgb(1,1,0)
    ,rgb(1,0,0)
    ,rgb(128.0/255,0,0)
  );

picture bar;
bar.unitsize(unit);

bounds range=image(v, (0,0),(n,m),Palette);

palette(bar,range,(0,0),(5,n),Right,Palette,
  PaletteTicks(scale(1)*"$%0f$",pTick=black+0.5pt,Step=1,beginlabel=false)
);

xaxis(0,m,RightTicks(scale(1)*"$%0f$",Step=20,step=10,beginlabel=false,black+0.5pt));
yaxis(0,n,LeftTicks(scale(1)*"$%0f$",Step=20,step=10,black+0.5pt));

add(bar.fit(),point(E),E);