Creating performance profiles

A performance profile is a kind of plot used in the optimization community; here below is an example from http://www.gamsworld.com/performance.

For an introduction, see Section 22.4 of Higham and Higham's book Matlab guide. Informally speaking, the graph reports which percentage (on the y axis) of a number of experiments done with competing methods is within a certain threshold (on the x axis) of the best result.

A formal specification is as follows.

• Suppose you run m tests to compare n existing methods. Let r_{ij} be the result (error, computational time... whatever; let's say that lower is better) obtained by method j on test i.

• The performance of method j on test i is defined as the ratio between its result and the best result on the same test, p_{ij} = r_{ij} / (min_j r_{ij}). Note that p_{ij} \geq 1.

• The performance function of method j is the function f_j(x) = 1/m * (number of tests i such that p_{ij} \leq x), that is, the "probability" that the method j is within a factor x of the best possible method. Note that f_j(x) is (weakly) monotonically increasing in x for each j.

• A performance profile is the plot of all functions f_j, as in the image above.

Do you know if there is support for creating performance profiles in tex? It is a very simple plot of course, so one in principle could preprocess the data and return a set of (x,y) pairs ready for plotting, but it would cut one step if the plot logic could be implemented natively in tex.

I have found nothing in the manual of pgfplots, which was my no. 1 candidate.

If someone wishes to implement it, here is a set of sample data, in pgfplots table format (method in columns, different tests in rows):

table{
ode23    ode45   ode113
1.26e-2  6.20e-3 1.56e-2
2.41e-1  1.53e-1 1.97e-1
3.74e-2  5.00e-2 6.68e-2
3.37e0   6.45e0  7.86e0
1.44e-1  1.56e-1 3.76e-2
5.06e-1  1.07e0  1.5e0
}


The output should look more or less like this image from the Higham & Higham book:

• I think we will need some idea of the nature of the raw input here. As you say, the way the example plot appears seems just to use a list of value pairs, for which plotting is pretty trivial. – Joseph Wright Aug 21 '14 at 8:47
• And beside of the input you should tell us an exemplary formula which you want to plot. I think just very few people are having time to read about performance plots in order to help you with this. Tell us a raw data example and a specific formula and I guess you will get a nice plot soon... Thanks. – LaRiFaRi Aug 21 '14 at 8:50
• I did not put an exact description because my question was "is there a package that does them out-of-the-box?". Now I realize that someone probably plans to quickly implement one in an answer, which would be awesome --- I love how everyone in this community really does a lot of work to help people. I'll edit my post adding formulas. – Federico Poloni Aug 21 '14 at 9:07

You are clearly a mathematician or something pretty close since your sentences are always inverted in an (ε, δ) way. You have to read the math backwards but the sentence forwards to decode :P

\documentclass[]{article}
\usepackage{pgfplotstable}
\pgfplotsset{compat=1.10,
perf name table/.default=mytable}
ode23    ode45   ode113
1.26e-2  6.20e-3 1.56e-2
2.41e-1  1.53e-1 1.97e-1
3.74e-2  5.00e-2 6.68e-2
3.37e0   6.45e0  7.86e0
1.44e-1  1.56e-1 3.76e-2
5.06e-1  1.07e0  1.5e0
}\mytable

\pgfplotstablegetcolsof{\mytable}% How many columns
\edef\mylastj{\number\numexpr\pgfplotsretval-1}% From zero to this num

%=========== Get the min of rows into a separate column =====================
\pgfplotstablecreatecol[
create col/assign/.code={%
\def\myrowmin{1e10}% Some big number that will be ignored in any min() invocation
\pgfplotsforeachungrouped \x in {0,...,\mylastj}{%
\pgfkeys{/pgf/fpu,/pgf/fpu/output format=sci}% To handle big numbers
\pgfmathparse{min(\myrowmin,\thisrowno{\x})}%
\pgfkeys{/pgf/fpu=false}%
\let\myrowmin=\pgfmathresult%
}%
\pgfkeyslet{/pgfplots/table/create col/next content}\myrowmin%
}
]{minperf}\mytable

%=========== Get each performance into a separate column =====================
\pgfplotsinvokeforeach{0,...,\mylastj}{
\pgfplotstablecreatecol[
create col/assign/.code={%
\pgfmathparse{\thisrowno{#1}/\thisrow{minperf}}%
\pgfkeyslet{/pgfplots/table/create col/next content}\pgfmathresult%
}
]{perf-#1}\mytable
}
\makeatletter
\pgfmathdeclarefunction{colleqx}{2}{%
\begingroup%
\c@pgf@countd=0% No rows satisfy yet
\pgfmathfloattoint{#1}%
\pgfplotstableforeachcolumnelement{perf-\pgfmathresult}\of\mytable\as\mycompval{%
\pgfmathfloatparsenumber{\mycompval}%
\pgfmathfloatlessthan{\pgfmathresult}{#2}%
}%
\pgfplotstablegetrowsof{\mytable}%
\pgfmathparse{\c@pgf@countd/\pgfplotsretval}%
\pgfmath@smuggleone\pgfmathresult%
\endgroup%
}

\begin{document}\noindent
\pgfplotstabletypeset[]{\mytable} % Let's see if minperf and perffun-i works?

\vspace{1cm}
\begin{tikzpicture}
\begin{axis}[title=Performances,height=6cm]
\addplot+[const plot] table[,x expr=\coordindex, y=perf-0] {\mytable};
\addplot+[const plot] table[,x expr=\coordindex, y=perf-1] {\mytable};
\addplot+[const plot] table[,x expr=\coordindex, y=perf-2] {\mytable};
\legend{ode23,ode45,ode113}
\end{axis}
\end{tikzpicture}

\begin{tikzpicture}
\begin{axis}[title=Performance Functions,height=6cm,
legend pos=outer north east,grid=both,no marks]
\legend{ode23,ode45,ode113}
\end{axis}
\end{tikzpicture}

\end{document}


This code leads to the following result

Some remarks;

1. What I did is first compute the performances and add them as columns as you can see from the first table.

2. Then I wrote a small math function which hides the table access macros and counting stuff such that it looks like a math function when pgfplots calls it but it is actually a counting function within each column. When you call it colleqx(<column index>,<x value>) (for number of rows of column index than or equal to x) it would give you the formula you provided for j = <column index>. Index numbering starts at zero.

3. The whole mechanism is based on the table name \mytable. Because otherwise I know that I would spend a whole evening on it. I have to leave it to you :) However it works with arbitrary number of cols and rows (I hope).

It always breaks my heart to see someone as proficient as N. Higham spends energy on things like matlab but not a proper programming language. Anyway, I digress.

• Awesome, thanks! I'll try to understand your code, the structure looks clear. Yes, indeed I am a mathematician, spotted. :) – Federico Poloni Aug 21 '14 at 16:43
• @percusse: Or you could use the built-in performance plot type, of course. ;-) – Jake Aug 21 '14 at 18:34
• @Jake To be honest for a few seconds I thought about opening the manual haha. – percusse Aug 21 '14 at 19:03