I must be missing something really simple and obvious, because I think I have done this many times before. A simple tryout file:
\documentclass[12pt]{article}
\usepackage[no-math]{fontspec}
\setmainfont{Lato}
\begin{document}
\Huge
this should be Lato --- and it is.
\fontfamily{Lato}\fontsize{30}{45}\selectfont
verbatim, this should be Lato, but instead is 'LMRoman17-Regular'.
\fontfamily{Ubuntu}\fontsize{30}{45}\selectfont
verbatim this should be Ubuntu font, but also is LMRoman17-Regular.
\bigskip
\bigskip
\bigskip
\scriptsize
poppler pdffonts:
\begin{verbatim}
UOTJVU+Lato-Regular CID TrueType Identity-H yes yes yes 4 0
KYEUXJ+LMRoman17-Regular CID Type 0C Identity-H yes yes yes 5 0
\end{verbatim}
\end{document}
LuaLaTeX picks up the \setmainfont
on Lato perfectly, but the hand-set fonts do not work. (Lato and Ubuntu are installed in macOS' Font book.)
The output is: