I am currently using a combination of mathpazo and mathspec for my maths typesetting. To be specific, my preamble contains the following:
\usepackage{amssymb}
\usepackage{mathrsfs}
\usepackage{mathpazo}
\usepackage{mathtools}
\usepackage{mathspec}
\usepackage{xltxtra,xunicode}
\defaultfontfeatures{Mapping=tex-text}
\setmainfont[Numbers=OldStyle]{Palatino Linotype}
\setsansfont{Optima}
\setmonofont{Consolas}
\setmathsf{Optima}
\setmathfont(Greek){Palatino Linotype}
\exchangeforms{phi}
However, I would like the option of using, say, the AMS blackboard bold fonts for mathbb, or the ones provided by STIX (or any other Unicode maths font) instead of the ones provided by mathpazo. So far nothing I've tried with \setmathbb has worked.
I have a similar problem with \setmathcal, and I notice that if I use calrsfs instead of mathrsfs there is no effect whatsoever.