# Having trouble with vowel positioning in Ezra SIL, XeLaTeX

I'm having trouble matching the example given in the answer What's a simple way to typeset a small amount of Hebrew content using TeX?.

I copied/pasted the example exactly as given in that answer, but the result comes out like this:

I've marked with red arrows the problem I'm experiencing, namely, the incorrect positioning of the vowel hireq under yodh (יִ): the vowel is way too high.

I'm using an up-to-date installation of TeX Live 2013 on Mac OS X Mavericks. The Ezra SIL font is version 2.51.

If I copy/paste the Hebrew text into Mac TextEdit and select Ezra SIL as the font, the hireq is positioned correctly.

Or, alternatively, if I switch font to SBL Hebrew:

``````\newfontfamily\hebrewfont[Script=Hebrew]{SBL Hebrew}% or whatever
``````

the hireq is also positioned correctly.

So it seems Ezra SIL was working for someone else before, but isn't now for me. Any ideas?

Could it be related to this issue (i.e., a consequence of the switch to HarfBuzz): Wrong vertical placement of bottom vowel signs (nikkud) in Hebrew for short and tall letters?

For reference, the output from XeTeX looks like this:

``````Running `LaTeX' on `testhebvowels' with ``xelatex  -interaction=nonstopmode "\input" testhebvowels.tex''
This is XeTeX, Version 3.1415926-2.5-0.9999.3 (TeX Live 2013)
restricted \write18 enabled.
entering extended mode
LaTeX2e <2011/06/27>
Babel <3.9f> and hyphenation patterns for 78 languages loaded.
(./testhebvowels.tex
(/usr/local/texlive/2013/texmf-dist/tex/latex/base/article.cls
Document Class: article 2007/10/19 v1.4h Standard LaTeX document class
(/usr/local/texlive/2013/texmf-dist/tex/latex/base/size12.clo))
(/usr/local/texlive/2013/texmf-dist/tex/latex/polyglossia/polyglossia.sty
(/usr/local/texlive/2013/texmf-dist/tex/latex/etoolbox/etoolbox.sty
(/usr/local/texlive/2013/texmf-dist/tex/latex/etex-pkg/etex.sty))
(/usr/local/texlive/2013/texmf-dist/tex/latex/makecmds/makecmds.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/xkeyval/xkeyval.sty
(/usr/local/texlive/2013/texmf-dist/tex/generic/xkeyval/xkeyval.tex
(/usr/local/texlive/2013/texmf-dist/tex/generic/xkeyval/keyval.tex)))
(/usr/local/texlive/2013/texmf-dist/tex/latex/fontspec/fontspec.sty
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/expl3.sty
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3names.sty
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3bootstrap.sty))
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3basics.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3expan.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3tl.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3seq.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3int.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3quark.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3prg.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3clist.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3token.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3prop.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3msg.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3file.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3skip.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3keys.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3fp.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3box.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3coffins.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3color.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3luatex.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3kernel/l3candidates.sty)
(/usr/local/texlive/2013/texmf-dist/tex/generic/oberdiek/ifpdf.sty))
(/usr/local/texlive/2013/texmf-dist/tex/latex/l3packages/xparse/xparse.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/fontspec/fontspec-patches.sty
(/usr/local/texlive/2013/texmf-dist/tex/latex/base/fixltx2e.sty)
*************************************************
* LaTeX warning: "xparse/redefine-command"
*
* Redefining document command \oldstylenums with arg. spec. 'm' on line 128.
*************************************************
) (/usr/local/texlive/2013/texmf-dist/tex/latex/fontspec/fontspec-xetex.sty
(/usr/local/texlive/2013/texmf-dist/tex/latex/base/fontenc.sty
(/usr/local/texlive/2013/texmf-dist/tex/latex/euenc/eu1enc.def)
(/usr/local/texlive/2013/texmf-dist/tex/latex/euenc/eu1lmr.fd))
(/usr/local/texlive/2013/texmf-dist/tex/xelatex/xunicode/xunicode.sty
(/usr/local/texlive/2013/texmf-dist/tex/latex/tipa/t3enc.def
(/usr/local/texlive/2013/texmf-dist/tex/latex/euenc/eu1lmss.fd))
(/usr/local/texlive/2013/texmf-dist/tex/latex/graphics/graphicx.sty
(/usr/local/texlive/2013/texmf-dist/tex/latex/graphics/graphics.sty
(/usr/local/texlive/2013/texmf-dist/tex/latex/graphics/trig.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/latexconfig/graphics.cfg)
(/usr/local/texlive/2013/texmf-dist/tex/xelatex/xetex-def/xetex.def))))
(/usr/local/texlive/2013/texmf-dist/tex/latex/fontspec/fontspec.cfg)))
(/usr/local/texlive/2013/texmf-dist/tex/generic/oberdiek/ifluatex.sty))
(/usr/local/texlive/2013/texmf-dist/tex/latex/polyglossia/gloss-hebrew.ldf
(/usr/local/texlive/2013/texmf-dist/tex/xelatex/bidi/bidi.sty
****************************************************
*
* bidi package (Bidirectional typesetting in
* plain TeX and LaTeX, using XeTeX engine)
*
* Description: A convenient interface for
* typesetting bidirectional texts in plain TeX
* and LaTeX. The package includes adaptations
* for use with many other commonly-used packages.
*
*
* 2013/11/16, v14.4
*
* 1.3c or higher (your choice)
*
* Location on CTAN: /macros/xetex/latex/bidi
*
* Issue tracker: https://github.com/vafa/bidi/issues
*
* Support: persian-tex@tug.org
****************************************************
(/usr/local/texlive/2013/texmf-dist/tex/generic/iftex/iftex.sty)
(/usr/local/texlive/2013/texmf-dist/tex/xelatex/bidi/biditools.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/oberdiek/zref-abspage.sty
(/usr/local/texlive/2013/texmf-dist/tex/latex/oberdiek/zref-base.sty
(/usr/local/texlive/2013/texmf-dist/tex/generic/oberdiek/ltxcmds.sty)
(/usr/local/texlive/2013/texmf-dist/tex/generic/oberdiek/infwarerr.sty)
(/usr/local/texlive/2013/texmf-dist/tex/generic/oberdiek/kvsetkeys.sty
(/usr/local/texlive/2013/texmf-dist/tex/generic/oberdiek/etexcmds.sty))
(/usr/local/texlive/2013/texmf-dist/tex/generic/oberdiek/kvdefinekeys.sty)
(/usr/local/texlive/2013/texmf-dist/tex/generic/oberdiek/pdftexcmds.sty)
(/usr/local/texlive/2013/texmf-dist/tex/latex/oberdiek/auxhook.sty))
(/usr/local/texlive/2013/texmf-dist/tex/generic/oberdiek/atbegshi.sty))
(/usr/local/texlive/2013/texmf-dist/tex/xelatex/bidi/latex-xetex-bidi.def
(/usr/local/texlive/2013/texmf-dist/tex/xelatex/bidi/footnote-xetex-bidi.def)
(/usr/local/texlive/2013/texmf-dist/tex/xelatex/bidi/article-xetex-bidi.def)
(/usr/local/texlive/2013/texmf-dist/tex/xelatex/bidi/graphicx-xetex-bidi.def))
(/usr/local/texlive/2013/texmf-dist/tex/latex/polyglossia/cal-util.def))
(/usr/local/texlive/2013/texmf-dist/tex/latex/polyglossia/babel-hebrewalph.def)) (/usr/local/texlive/2013/texmf-dist/tex/latex/polyglossia/gloss-english.ldf)
(./testhebvowels.aux)
(/usr/local/texlive/2013/texmf-dist/tex/latex/tipa/t3cmr.fd)
Setting \if@calendar@hebrew

LaTeX Font Warning: Font shape `EU1/EzraSIL(0)/bx/n' undefined
(Font)              using `EU1/EzraSIL(0)/m/n' instead on input line 15.

[1] (./testhebvowels.aux)

LaTeX Font Warning: Some font shapes were not available, defaults substituted.

)
Output written on testhebvowels.pdf (1 page).
Transcript written on testhebvowels.log.

LaTeX finished at Fri Jan 31 10:37:33
``````
• The HarfBuzz bug report you link to is exactly the problem. So you have answered your own question. :) And since the conclusion of the HarfBuzz folk is that the problem is font dependent, and therefore not their problem and since XeLaTeX now uses HarfBuzz, I don't see any solution other than to find a font that works. – Alan Munn Jan 31 '14 at 3:32
• Your analysis of the situation seems to be exactly right. And yet, it can't be the whole story, since XeLaTeX did use to display the text correctly with Ezra SIL. So the denial of responsibility by the HarfBuzz people does not seem to be completely reasonable. – Richard Walker Jan 31 '14 at 8:21
• It is the whole story, to the extent that XeLaTeX made the decision to use HarfBuzz as the shaping engine, replacing the ICU engine that was no longer maintained and buggy. The fact that the older version was able to deal with the problematic font has to be weighed against the many other problems the ICU engine was causing. See khaledhosny.org/node/198 for some more details. – Alan Munn Jan 31 '14 at 9:52

Update: I’m no longer sure that what I wrote below is correct, namely this specific composition does not seem to be part of Unicode normalisation and this seems to be a HarfBuzz “feature” (or misfeature for the case of Ezra SIL).

This is a font bug. The font has a `U+FB1D` HEBREW LETTER YOD WITH HIRIQ character whose glyph has the hirq positioned high. This worked with pre-HarfBuzz XeTeX because the text actually uses `U+05D9` HEBREW LETTER YOD + `U+05B4` HEBREW POINT HIRIQ and those are positioned correctly using OpenType anchors, but it fails with HarfBuzz because `U+FB1D` is canonically equivalent to `U+05D9` + `U+05B4` so HarfBuzz applies Unicode canonical composition here and uses the glyph of the `U+FB1D`.

Here is a comparison between XeTeX 0.9998 (pre-HarfBuzz) and XeTeX 0.9999 (HarfBuzz), as you can see the pre-HarfBuzz XeTeX was producing different output for canonically equivalent input.

HarfBuzz behavior is reasonable as Unicode mandates that canonical equivalent sequences should be treated the same, so the user should get the same output whether the input was `U+FB1D` or `U+05D9` + `U+05B4`, but Ezra SIL produces different output (if normalization was not performed), so it is the one at fault here.

I suggest you report this to Ezra SILL developers since the issue is not limited to XeTeX or HarfBuzz (basically any application applying NFC normalization, or where input uses `U+FB1D` directly, will be affected).

One workaround to this is to edit the font and drop the `U+FB1D` character (either by deleting its glyph or hiding it from the `cmap` table), in which case HarfBuzz will not apply the composition here (and if one inputs `U+FB1D` it will decompose it since the font does not have it, so you get improved result for this case as well, compared to pre-HarfBuzz XeTeX).

This is entirely different from the HarfBuzz bug report you found, as it involves legacy not-OpenType fonts where HarfBuzz will try to position the marks itself using heuristics that will not always produce the most optimal positioning.

• Thank you for putting me on to the composite glyph FB1D. I have now opened the font using fontforge and I can confirm that glyph FB1D does indeed contain this composite glyph with incorrectly placed hireq. I edited the glyph by lowering the placement of the hireq. The modified font works correctly. – Richard Walker Jan 31 '14 at 22:38
• However, I then tried a second experiment as you suggested: removing the glyph FB1D from the font. The result is the same as the original: a misplaced hireq! So it seems Ezra SIL is a "legacy not-OpenType font" affected by the HarfBuzz issue. – Richard Walker Jan 31 '14 at 22:45
• The font SBL Hebrew (TTF) also has a composite glyph FB1D, but with correctly-positioned hireq. So ... it seems the font designers have put this composite glyph into the font precisely as a workaround. It's just that the designer(s) of Ezra SIL left the hireq in the wrong place :-( – Richard Walker Jan 31 '14 at 22:50
• I want to backtrack on my last two comments. I've been playing with the version of HarfBuzz included in XeTeX as a standalone library (using the hb-view utility) and it's no longer clear to me that HarfBuzz per se is necessarily at fault: it could be related to how it was integrated into XeTeX (selection of #defines, etc.). – Richard Walker Feb 1 '14 at 6:00
• Conversely, as for the "misplaced" hireq in composite glyph FB1D, I may have been harsh on Ezra SIL. HarfBuzz has specific code to compose 05D9+05B4 and display glyph FB1D (see comment in the source code referring to "old fonts"), but this seems (to me) against the spirit of the Unicode spec referring to canonical forms, and, although it may be a good thing to do for "old fonts", unfortunately, it turns out to be precisely the wrong thing to do for Ezra SIL. (But only once we sort out the other issue!) – Richard Walker Feb 1 '14 at 6:01

I'm not sure what are the availability on mac, but I use Culmus David font on Linux in order to typeset hebrew with XeTeX, and it's works grate:

The fonts can be downloaded in a free license from http://culmus.sourceforge.net/ . I hope they works proparly on mac.