# Bidi algorithm and xelatex

Is there a way to make xelatex follow the standard bidi algorithm?

I'm using setup similar to this:

\documentclass[12pt]{article}
\TeXXeTstate=1
\usepackage{polyglossia}
\setmainfont{Linux Libertine O}
\setsansfont{Linux Biolinum O}
\setdefaultlanguage{english}
\setotherlanguages{russian,hebrew}
\usepackage{bidi}


Now everything is fine and dandy, except that I have to use \begin{hebrew} and \end{hebrew} or directionality commands to mark up Hebrew passages. This is rather annoying.

One would expect that \usepackage{bidi} would make xelatex follow the standard bidi algorithm when deciding on text direction, but apparently this doesn't happen.

Is there any other package that enables the bidi algorithm, or provides some other way to make xelatex infer text direction from the script? If not, perhaps there's some technical or philosophical issue I'm not aware of that prevents this?

• If you use vim, I have a script that does the wrapping for you. – A Gold Man Nov 25 '17 at 21:03
• @AGoldMan I do use vim though IME it's not the greatest editor for bidi text. Still I'd like to try the script. – n.m. Nov 26 '17 at 7:02
• There is an experimental package called unicode-bidi. I have not yet had success getting it to work without xepersian, but maybe it's a good place to start. – David Purton Nov 26 '17 at 9:52
• @n.m. If you run vim in mlterm, then it's pretty solid. You can check my script at my github. I don't know if it'll be good for your use case. – A Gold Man Nov 26 '17 at 11:53

# Using lualatex and babel

I recently discovered that babel use basic support for this using lualatex. You still have to use \selectlanguage{hebrew} to switch to a RTL paragraph, but inline bidi seems to work OK.

\documentclass{article}
\usepackage[nil,bidi=basic]{babel}
\babelprovide[import=en-AU,main]{australian}
\babelprovide[import=he]{hebrew}
\babelfont{rm}{Linux Libertine O}
\babelfont{sf}{Linux Biolinum O}
\pagestyle{empty}
\begin{document}
In the beginning God created the heavens and the earth. בְּרֵאשִׁית בָּרָא אֱלֹהִים אֵת
הַשָּׁמַיִם וְאֵת הָאָרֶץ. (Genesis 1:1)

\selectlanguage{hebrew}
בְּרֵאשִׁית בָּרָא אֱלֹהִים אֵת הַשָּׁמַיִם וְאֵת הָאָרֶץ (Genesis 1:1)
\end{document}


# Using xelatex and polyglossia

Here's an example which uses the same technique as unicode-bidi. It's not very robust, and is really only good for inline Hebrew, not paragraph Hebrew.

\documentclass{article}
\usepackage{polyglossia}
\setmainlanguage{english}
\setotherlanguage{hebrew}
\newfontfamily{\hebrewfont}{Linux Libertine O}[Script=Hebrew]
\makeatletter
\newcount\unicode@bidi@hebrew@alphabets
\newXeTeXintercharclass\unicode@bidi@hebrew@alphabets@charclass
\unicode@bidi@hebrew@alphabets=\^^^^0591 \loop \XeTeXcharclass
\unicode@bidi@hebrew@alphabets \unicode@bidi@hebrew@alphabets@charclass
\XeTeXinterchartoks \z@ \unicode@bidi@hebrew@alphabets@charclass {\unicode@bidi@starthebrew}
\XeTeXinterchartoks \e@alloc@intercharclass@top \unicode@bidi@hebrew@alphabets@charclass {\unicode@bidi@starthebrew}
\XeTeXinterchartoks \e@alloc@intercharclass@top \z@ {\unicode@bidi@finishhebrew}
\XeTeXinterchartoks \unicode@bidi@hebrew@alphabets@charclass \z@ {\unicode@bidi@finishhebrew}
\newcommand*{\unicode@bidi@starthebrew}{\if@nonlatin\else\bgroup\beginR\hebrewfont\@nonlatintrue\fi}
\newcommand*{\unicode@bidi@finishhebrew}{\if@nonlatin\unskip\endR\egroup{ }\fi}
\makeatother
\setlatin
\pagestyle{empty}
\begin{document}
In the beginning God created the heavens and the earth. בְּרֵאשִׁית בָּרָא אֱלֹהִים אֵת
הַשָּׁמַיִם וְאֵת הָאָרֶץ׃ (Genesis 1:1).
\end{document}


• Thank you. It looks like a robust implementation would be really difficult. – n.m. Nov 26 '17 at 15:20

XeTeX deals with the typesetting word by word. Within each word, it'll use the normal unicode bidi-algorithm, but outside of that it treats each word seperately, meaning that after a word is put into the horizontal list being worked on, it just becomes a "box" (but not a real box accessible from \lastbox) with dimensions.

This being the case, it would be impossible to properly implement the bidi algorithm, because it assigns directionality based on the contents of the line, but once TeX is up to line breaking, it no longer knows what the contents of any "box" are.

This itself is actually rather convenient as far as the system goes, because faults in the bidi algorithm, at least as applied to the specific dialect of hebrew I write in, were (partially) the reason I switched to TeX in the first place.

With this system, you'll never have backwards parentheses where you don't expect them, because you told TeX when they should be RTL or not. When the user has complete control over the directionality, that allows the text to come out as expected.

A solution could be cooked up using \XeTeXinterchartoks, but that would be involved and depend on your use case.

• The engine could insert implicit directionality commands between words based on the script. This only requires limited look-ahead. I know the bidi algorithm cannot cope with all use cases but we have bidi override for these cases. Thank's for the XeTeXinterchartoks suggestion, I'll look at it. – n.m. Nov 26 '17 at 7:00