It's not clear whether
\XeTeXglyph1 is the first glyph in the font. Where does the glyph count start at (0 or 1)?
XeTeX uses the regular scheme for numbering glyphs from OpenType fonts which is also used by other programs, the first glyph is
But the first glyph by definition is the
.notdef glyph which is used as replacement if a glyph is missing. So if you actually try
\XeTeXglyph0 it will look as if you used a character the font doesn't support. The first real glyph is at index