This article says xetex can only find fonts installed by texlive with its filename instead of font name, because xetex can only find fonts by font name in the system's font system.
And here says with the command texlive-enable-fontconfig, texlive will add the its font to the system fontconfig.
The result is adding a file: /etc/fonts/conf.d/09-texlive.conf
However, I still cannot find the font by font name when using xetex. (If using font filename, it's fine.)