Here my personal experience writing my thesis: Using one (serif) font in the text, pictures and charts seems to be the most elegant choice. However, using sans serif in diagrams, plots, figures, ... looks more technical and can be better legible. (all as you already mentioned)
I actually wanted to make the text of the captions of the figures and tables to be easily distinguishable from the main text. So I used a sans serif font for all tables and figures including their captions with one step smaller font size (
\small). I made sure that all sans serif elements have the same font size. I also adjusted the leading to give similar blackness levels on the paper for the serif main text and the sans serif captions. Robert Bringhurst used a similar approach in his typography book.
One possible issue: If the sans serif font has less font features (e.g. small caps) then the serif font, one has to find a compromise...
Even if you can only adjust the figure's font, the technical, more clear appearance seems preferable to me.