Summarising the various comments as an answer, the bottom line is that a target of 5 MB is unrealistic. For example, the Windows XeTeX binary on my system is 2.7 MB, while as Ulrike says
The MiKTeX package with XeTeX binaries has a size of 14MB.
(On my Unix system, the XeTeX binary is 17.8 MB). You need of course in addition to XeTeX the
xdvipdfm system (I'm not sure exactly how that is packaged up on different systems.)
As an absolute minimum, you need in addition to the XeTeX binary some structure for finding files (
kpsewhich or MiKTeX's equivalent), basic fonts (Computer Modern and Latin Modern), then a basic LaTeX (the base/required/tools, plus whatever you feel is 'minimal'). That is all going to add up.
Looking at the existing small distributions, BasicTeX (Mac-only) is 92 MB and MiKTeX basic is 157 MB. You could try to build something like one of these yourself, but the most obvious way to proceed is
Get either MiKTeX portable or TeX Live portable, install it and then
remove the things you don't need (in MiKTeX you can use the package
manager to remove packages).
(again from Ulrike's comment).
To get the size down, you should also look to remove all of the
doc tree, as this tends to be quite big. Of course, you'll be left with no documentation, but it will be smaller.