# pgfplots: Reshape a MATRIX of data before conditional 3D surface plotting

As far as I know, for plotting a surface using pgfplots, the data should be organized as follows:

x1 y1 z11
x1 y2 z12
x1 y3 z13
.  .   .
.  .   .
.  .   .
xn ym znm


However, usually, I get the data to be plotted in the following matrix shape

   y1  y2  y3   ...      ym
x1 z11 z12 z13  ...     z1m
x2 z21 z22 z23           .
x3 z31 z32 z33           .
.   .            .       .
.   .              .
.   .                .
xn zn1       ...        znm


So, is there any way to reshape or use the data as it is for plotting without having to manually organize it the way pgfplots expects?

This is an exemplary sample of data set to be conditionally surface plotted the same way done in the answer to my previous question.

     0.8    0.78    0.76    0.74    0.72
3.1 12.17   11.5    100     100     100
3.3 12.17   11.5    11.5    100     100
3.5 12.17   11.5    11.5    11.5    100
3.7 12.17   11.5    11.5    11.5    11.5
3.9 12.17   11.5    11.5    11.5    11.5
4.1 12.17   11.5    11.5    11.5    11.5


## Edit 1

As suggested by Christian Feuersänger, I made a script by MATLAB as follows to reshape the data, but I don't have any idea how to make its corresponding lua syntax.

Data=[
12.17   11.5    100     100     100;
12.17   11.5    11.5    100     100;
12.17   11.5    11.5    11.5    100;
12.17   11.5    11.5    11.5    11.5;
12.17   11.5    11.5    11.5    11.5;
12.17   11.5    11.5    11.5    11.5;
];

xvalues=3.1:0.2:4.1;
yvalues=0.8:-0.02:0.72;

ReshapedData=zeros(3,numel(Data)); % x y z

fileID = fopen('file.txt','w');% file to save data

for nx=1:length(xvalues)
for my=1:length(yvalues)
colno=my+(nx-1)*length(yvalues);
ReshapedData(1,colno)=xvalues(nx);
ReshapedData(2,colno)=yvalues(my);
ReshapedData(3,colno)=Data(nx,my);
fprintf( fileID,'%.1f\t%.2f\t%.2f\r\n',ReshapedData(:,colno) );
end
fprintf( fileID,'\r\n');
end
fclose(fileID);

• You will need some scripting solution to reshape the data. This can be an external tool (like a python script) or an embedded script. Standard embedded scripting solutions in latex are: plain TeX (quite uncommon language and bad data structure support) or lua. Embedding scripting solutions with lua require you to invoke lualatex instead of pdflatex. If you come up with a lua script, we can help you to inject the result into \pgfplotstableread. – Christian Feuersänger Jan 20 '18 at 11:49
• @ChristianFeuersänger Thanks for your suggestion. I have already made a script using MATLAB (Edit 1) since it is the most known language to me. So, i would be grateful if you could help me convert it to lua. – Diaa Jan 20 '18 at 12:29
• For the record: I have added a Feature Request for that to the PGFPlots Tracker. – Stefan Pinnow Jan 20 '18 at 15:48

As pointed out in some comments, this is possible with lua.

My script assumes data ordered as follows, whitespace-delimited only. This is basically as asked in the question. The indentation in the first line is optional.

   y1  y2
x1 z11 z12
x2 z21 z22
x3 z31 z32


The script reads the data, stores it in tables and writes the tables in an output file such that the data is reformated in the desired way.

x1 y1 z11
x1 y2 z12
x2 y1 z21
x2 y2 z22
x3 y1 z31
x3 y2 z32


A minimal example code, where the input file with your example data and the lua script are included and written into files with the help of filecontents, reads as follows.

\documentclass{article}
\usepackage{pgfplots,filecontents,luacode}

\begin{filecontents*}{mat-in.txt}
0.8    0.78    0.76    0.74    0.72
3.1 12.17   11.5    100     100     100
3.3 12.17   11.5    11.5    100     100
3.5 12.17   11.5    11.5    11.5    100
3.7 12.17   11.5    11.5    11.5    11.5
3.9 12.17   11.5    11.5    11.5    11.5
4.1 12.17   11.5    11.5    11.5    11.5
\end{filecontents*}

\begin{filecontents}{mat-rewrite.lua}
\begin{luacode}

function rewrite_mat(infilename, outfilename)
infile  = io.open( infilename, "r")
outfile = io.open(outfilename, "w")

ynum = 0
xnum = -1
yvals={}
xvals={}
zvals={}

for line in infile:lines() do
t={}                               -- make a table from the line (whitespace-delimited)
length=1
for v in string.gmatch(line, "%S+") do
t[length]=v
length = length + 1
end

xnum = xnum + 1                    -- count number of x values

if xnum < 1 then                   -- first line: count number of y values
ynum  = length -1
yvals = t
else
xvals[xnum]=t                 -- all other lines: store x and z values
for y=1,ynum do
index = ynum*(xnum-1)+y
zvals[index]=t[y+1]
end
end
end

infile:close()

for x = 1,xnum do
for y = 1,ynum do
outfile:write(xvals[x] .. " " .. yvals[y] .. " " .. zvals[(x-1)*ynum+y] .. " " .. "\string\n")
end
end

outfile:close()

end
\end{luacode}
\end{filecontents}

\input{mat-rewrite.lua}

% run function mat-in.txt for structure, mat-in2 for matrix with values
\directlua{rewrite_mat("mat-in.txt","out.txt")}

\begin{document}
\begin{tikzpicture}
\begin{axis}[mesh/ordering=y varies, unbounded coords=jump,]
table[
x index =0,
y index =1,
% use an expression to state that z values of 100 should be ignored
z expr={\thisrowno{2} == 100 ? NaN : \thisrowno{2}},
] {out.txt};
\end{axis}
\end{tikzpicture}
\end{document}


I have reused the code from the previous question/answer to drop the z=100 values. The output plot of this example looks not very spectacular... But it demonstrates the conversion from the matrix data to a vector and subsequent plotting. • +1 Very nice solution. As I stated in the comment below the question I made a Feature Request in the PGFPlots Tracker regarding this issue. Do you think you could generalize your solution? Or even better: Would you be able to implement this features into PGFPlots? – Stefan Pinnow Aug 6 at 17:15
• @Stefan Pinnow I can for sure try to integrate it but first i will generalize it.i have comment chars, custom table delimiters and a bit of configurability regarding table structure in mind, how x and y values are ordered around the matrix. Any other suggestions? – Faekynn Aug 6 at 21:15
• i should also directly extract mesh/rows and mesh/cols. – Faekynn Aug 6 at 21:16
• Many thanks for your willingness to implement that feature! I suggest that we continue the discussion about the implementation in the comments to the feature-request. It fits better there than here. About other suggestions: There could be the possibility that only the z-values of the matrix are given. If you have "moved" the comments to the tracker I will contact Christian (the author of PGFPlots) as well that he should comment on the needed features as well. He has -- of course -- a much deeper insight than me. – Stefan Pinnow Aug 7 at 4:33