Pgfplots comes with builtin support for single sort keys.
However, if is possible to plug in multiple sort keys -- with a bit of efford and inside knowledge:
\documentclass[a4paper]{standalone}
\usepackage{pgfplots}
\usepackage{pgfplotstable}
\begin{document}
\def\pgfplotsinvokeiflessthan#1#2#3#4{%
\pgfkeysvalueof{/pgfplots/iflessthan/.@cmd}{#1}{#2}{#3}{#4}\pgfeov
}%
\def\pgfplotsmulticmpthree#1#2#3#4#5#6\do#7#8{%
\pgfplotsset{string <}%
\pgfplotsinvokeiflessthan{#1}{#4}{%
% first key <:
#7%
}{%
\pgfplotsinvokeiflessthan{#4}{#1}{%
% first key >:
#8%
}{%
% first key ==:
\pgfplotsset{string <}%
\pgfplotsinvokeiflessthan{#2}{#5}{%
% second key <
#7%
}{%
\pgfplotsinvokeiflessthan{#5}{#2}{%
% second key >
#8%
}{%
% second key ==
\pgfplotsset{float <}%
\pgfplotsinvokeiflessthan{#3}{#6}{%
% third key <
#7%
}{%
% third key >=
#8%
}%
}%
}%
}%
}%
}%
\pgfplotstabletypeset[
create on use/sortkey/.style={
create col/assign/.code={%
\edef\entry{{\thisrow{Kind}}{\thisrow{Mtx}}{\thisrow{P}}}%
\pgfkeyslet{/pgfplots/table/create col/next content}\entry
}
},
sort key=sortkey,
sort cmp={%
iflessthan/.code args={#1#2#3#4}{%
\edef\temp{#1#2}%
\expandafter\pgfplotsmulticmpthree\temp\do{#3}{#4}%
},
},
sort,
columns/Mtx/.style={string type},
columns/Kind/.style={string type},
]{
Mtx Kind P Sp
A LP 16 4.2
C OPT 16 72.5
A LP 64 20.3
B OPT 16 5.7
B OPT 64 16.4
A LP 256 90.4
}
\end{document}

The approach consists of two parts: (1) the creation of a temporary column called sortkey
which expands to {A}{LP}{16}
for the first row and (2) the definition of a customized sort cmp
. The customized sort cmp
here uses a multi-key comparison (lexicographical compare) with string <
for the first two keys and float <
for the third.
Note that sort key=sortkey
instantiates the create on use
specification while sort cmp
defines the customized comparison.
The key sort
activates the sort functionality
The approach is fairly general and should probably become a general-purpose utility; I will think about a suitable predefined style.