You can't use every OpenType font as a math font for
unicode-math. At least a
MATH table in the font is necessary, along with a
math script. Although some of the basic setups can be performed, as it is shown by the italic letters in the formula, some important parameters for math typesetting are missing.
In your case the big problem is with the radical: as David Carlisle remarks, TeX uses for the thickness of the bar over the radicand the height of the radical symbol; however XeTeX and LuaTeX know how to use a different parameter from the MATH table, so the problem doesn't appear.1
A real math font must have also other information for the correct kerning of math characters in formulas. See the documentation of
mathspec for more details and, probably, many more glyphs than FreeSerif has.
If your text font is FreeSerif, then a good choice for the math font can be either XITS Math or TG Termes Math, both based on Times.
1 Unfortunately, due to a bug in the current version of LuaTeX, the big black blob can appear also if the math font has the correct parameters in the