I use fontspec with LuaLaTeX to load fonts from external file even if they are not installed on the system. Such as
\usepackage{fontspec}
\setmainfont[Path=fonts/,
Extension=.otf,
UprightFont=*-Regular,
ItalicFont=*-Italic,
BoldFont=*-Bold,
SmallCapsFont=*-AllSC,]{EBGaramond12}
I also use unicode-math, because it's awesome, usually with XITS Maths for the operators and numbers and EB Garamond for the letters.
\usepackage[math-style=TeX]{unicode-math}
\setmathfont{XITS Math}
\setmathfont[range=\mathbfsfit/{greek,Greek,latin,Latin}]{EB Garamond}
Is there a way to combine these packages to use the letters from the local EB Garamond instead of the system's one?

fontspecoptions are valid forunicode-mathas well. – Khaled Hosny Jan 22 at 14:54