XeTeX allows easy access to system fonts, but none of the standard formats (here the LaTeX format) load these. Indeed, it's not possible to save system fonts into a format: they have to be loaded when used. As such, when you run
xelatex you get almost the same preloaded code as you do with
pdflatex (there are a few changes to deal with assigning codes to an extended set of characters). The standard TeX fonts don't have coverage of all of Unicode, so the characters you want to print are simply not available.
It's not absolutely necessary to use
fontspec to use system fonts with XeLaTeX: you can load the font directly using the
\font primitive, which is extended by XeTeX. However, using
fontspec makes life a lot easier. With the standard settings,
fontspec will load the Latin Modern font, which includes a lot wider glyph coverage than LaTeX 'out of the box'. It's important to note that you will see only see the glyphs that the font includes: a classic issue is that most western fonts don't cover Chinese, etc., so you need an appropriate font here.