mathspec
is a clever attempt to make it possible to use math in XeLaTeX documents, with system fonts for the letters and symbols from standard math fonts.
With unicode-math
one can basically use only specially tailored OpenType math fonts such as Latin Modern Math, TeX Gyre Termes Math, TeX Gyre Pagella Math, XITS Math, Asana Math (among the free ones) or Cambria Math and Lucida Bright Math (not free).
mathspec
With this package you can do limited math with the main system font. Just a silly example:
\documentclass{article}
\usepackage{mathspec}
\setmainfont[Ligatures=TeX]{Old Standard}
\setmathsfont(Digits,Latin,Greek){Old Standard}
\begin{document}
Some text and a formula $a+b=\int_{\xi}^{\theta} f(x)\,dx$.
\end{document}

The emulated math is not bad, and the package has features for manually correcting the possible bad spacings, which happen because the fonts have really no support for math. For instance, inputting the formula as
$a+b=\int_{\xi}^{\theta} "f(x)\,dx$
would give the better result

The "
means ”add some space on both sides of the following letter”.
Digits, Latin and Greek letters can be taken from different fonts.
unicode-math
The unicode-math
is compatible both with XeLaTeX and LuaLaTeX. Here's the same example with Lucida Bright:
\documentclass{article}
\usepackage{unicode-math}
\setmainfont[Ligatures=TeX,Scale=0.85]{Lucida Bright OT}
\setmathfont[Scale=0.85]{Lucida Bright Math OT}
\begin{document}
Some text and a formula $a+b=\int_{\xi}^{\theta} f(x)\,dx$.
\end{document}

Spacing is correct without manual intervention, because the font used for math is a real math font. Here's the same with TeX Gyre Pagella:
\documentclass{article}
\usepackage{unicode-math}
\setmainfont[Ligatures=TeX]{TeX Gyre Pagella}
\setmathfont{TeX Gyre Pagella Math}
\begin{document}
Some text and a formula $a+b=\int_{\xi}^{\theta} f(x)\,dx$.
\end{document}

Unicode input
One can input math with Unicode symbols, but this is not mandatory. It's possible both with mathspec
and unicode-math
. For instance, the integral with mathspec
could be input as
$∫_ξ^θ "f(x)\,dx$
provided that ∫
is made known to the environment, for instance with
\usepackage{newunicodechar}
\newunicodechar{∫}{\int}
Some symbols (in particular letters) are already known.
Nothing particular is needed for inputting the integral as
$∫_ξ^θ f(x)\,dx$
with unicode-math
. However the traditional syntax is understood by the package.