Do you just want something like this?
EDIT: The following code is edited to eliminate loading charter
which turns out to be pointless. mathdesign
wants to set \rmdefault
to mdbch
rather than bch
- so let it.
\documentclass{article}
\usepackage[T1]{fontenc}
\renewcommand{\sfdefault}{fvs}
\renewcommand{\ttdefault}{fvm}
\usepackage[charter, sfscaled, ttscaled]{mathdesign}
\begin{document}
Oh sphinx of black quartz, hear my vow!
\textsf{Oh sphinx of black quartz, hear my vow!}
\texttt{Oh sphinx of black quartz, hear my vow!}
\[
E=m\mathrm{c}^2
\]
\[
\mathrm{roman\ maths}\ \mathsf{sans\ maths}\ \mathtt{typewriter\ maths}\ \mathbf{bold\ maths}
\]
\end{document}

XCharter
Thanks to Manuel's comment, I discovered a relatively new extension of Charter provided by the XCharter
package (yes, it is capitalised). Based on the package documentation, I tried this to adjust the sizing appropriately:
\documentclass{article}
\usepackage[T1]{fontenc}
\renewcommand{\sfdefault}{fvs}
\renewcommand{\ttdefault}{fvm}
\usepackage[charter, expert, sfscaled, ttscaled]{mathdesign}
\usepackage[scaled=.96,osf]{XCharter}% matches the size used in math
\linespread{1.04}
\begin{document}
Oh sphinx of black quartz, hear my vow! 0123456789
\textsf{Oh sphinx of black quartz, hear my vow! 0123456789}
\texttt{Oh sphinx of black quartz, hear my vow! 0123456789}
\[
E=m\mathrm{c}^2
\]
\[
0123456789
\]
\[
\mathrm{roman\ maths}\ \mathsf{sans\ maths}\ \mathtt{typewriter\ maths}\ \mathbf{bold\ maths}
\]
\textsc{\itshape Oh sphinx of black quartz, hear my vow! 0123456789}
\textsc{Oh sphinx of black quartz, hear my vow! 0123456789}
\textsc{\itshape\bfseries Oh sphinx of black quartz, hear my vow! 0123456789}
\textsc{\textbf{Oh sphinx of black quartz, hear my vow! 0123456789}}
\end{document}

It is worth noting that the manual doesn't really recommend using mathdesign
. For an alternative, see example 2 on page 2 of XCharter's manual.
mathunicode
?fontspec
? opentype versions of the fonts? – cfr Apr 8 '14 at 16:51[utf8]{inputenc}
rather than[utf8x]{inputenx}
, by the way. – cfr Apr 9 '14 at 21:34