After a lot of trying I found something that kind works:
\usepackage{fontspec}
\usepackage[math-style=upright]{unicode-math}
\setmainfont{calibri}[Ligatures=TeX, SmallCapsFeatures={Letters=SmallCaps,LetterSpace=5}, Path=fonts/, Extension = .ttf, UprightFont = calibri, BoldFont = calibrib, ItalicFont = calibrii, BoldItalicFont = calibriz]
\setmathfont{XITS Math}
\setmathfont{calibri}[
range={up/{greek,Greek,latin,Latin,num},
it/{greek,Greek,latin,Latin,num},
bfup/{greek,Greek,latin,Latin,num},
bfit/{greek,Greek,latin,Latin,num},
frak/{greek,Greek,latin,Latin,num}},
Path=fonts/, Extension=.ttf, UprightFont = calibri, BoldFont = calibrib, ItalicFont = calibrii, BoldItalicFont = calibriz
]
\DeclareMathOperator{\uphi}{\symbol{"03C6}}
But it's very unreliable. I need to load XITS Math for the symbols since a lot of them are missing in calibri, \mathbf does not work with a lot of character.
I would not feel confident to use this solution on a large scale document, seems kinda buggy.
\setmathfont{XITS Math}
for the math mode. – Sebastiano May 1 '19 at 20:35