How to force sans-serif fonts in matplotlib with xelatex PGF backend

Question

This is a slightly borderline question for matplotlib-xelatex combination.

I ma having an issue forcing a sans-serif font family in a png/pdf output of my matplotlib. I wanted to use tex rendering for consistency but for technical reasons wanted to output an png/pdf image rather than PGF output. Either way though fonts for labels, tick labels and legend are all serif even though I am trying to get sans-serif.

Here's a minimal example:

from numpy import linspace
import matplotlib
matplotlib.use('pgf')
from matplotlib.pyplot import close, figure, rcParams

params = {
    'font.family': 'sans-serif',
    'font.sans-serif': 'Linux Biolinum',
    'text.usetex': True,
    'pgf.texsystem': "xelatex",
    'pgf.rcfonts': True,
    'text.latex.preamble': [
        r'\usepackage{libertine}',
        r'\usepackage{fontspec}',
        r'\setmainfont{Linux Biolinum}',
    ]
}
rcParams.update(params)
my_figure = figure()
x = linspace(-1, 1)
axes = my_figure.gca()
axes.plot(x, x**2, label='\\textrm{Re}')
axes.legend(loc='upper left', handlelength=2)
my_figure.savefig(
    'test.png',
    box_inches='tight',
    dpi=300)
close(my_figure)