Inverted parentheses in RTL text in LuaLaTeX

Question

I want to typeset Hebrew (RTL) text (either within LTR paragraphs or in its own RTL paragraphs) in LuaLaTeX. I manage to do this well with XeLaTeX using the bidi package, but when I use LuaLaTeX’s bidi support (§2.3 in the LuaTeX manual) the parentheses in the RTL text are come out wrong, inverted.

For example, compare this good output:
of this file, compiled with XeLaTeX:

%! TEX program = xelatex
\documentclass{standalone}

\usepackage{fontspec, bidi}
\setmainfont{Alef}
\newcommand{\Heb}[1]{\RL{\fontspec[Script=Hebrew, Language=Hebrew]{Alef}#1}}

\begin{document}
Alpha (Beta) Gamma;
\Heb{אלף (בית) גימל};
A (B) C
\end{document}

… with this bad output:
of this file, compiled with LuaLaTeX:

%! TEX program = lualatex
\documentclass{standalone}

\usepackage{fontspec}
\setmainfont{Alef}
\newcommand{\Heb}[1]{\bgroup\textdir TRT\fontspec[Script=Hebrew, Language=Hebrew]{Alef}#1\egroup}

\begin{document}
Alpha (Beta) Gamma;
\Heb{אלף (בית) גימל};
A (B) C
\end{document}

It seems the bidi package knows to invert brackets within RTL text. How can I have the same effect with LuaLaTeX?

Many thanks! :-)

---

Update: David Purton’s answer seems to solve the problem. Is there a solution which doesn’t involve using babel?

Answer 1

You can use babel. As of version 3.19, you don't even have to mark up for simple things. See §1.27 of the babel manual.

%! TEX program = lualatex
\documentclass{standalone}
\usepackage[nil,bidi=basic]{babel}
\babelprovide[import=en-AU,main,mapfont=direction]{australian}
\babelprovide[import=he,mapfont=direction]{hebrew}
\babelfont{rm}[Ligatures=TeX]{Linux Libertine O}
\begin{document}
Alpha (Beta) Gamma;
אלף (בית) גימל;
A (B) C
\end{document}

Answer 2

To answer you update, to make the code you linked in the comments to work with latex you need to replace callback.register with luatexbase.add_to_callback, since latex does not support callback.register.

Another change is to replace the line if n.id == node.id("whatsit") and n.subtype == 7 then, since a dir node are not a whatsit node anymore.

The file char-def.lua cannot be found in the linked code anymore, but I found it here. Here is an example file:

\documentclass{article}
\directlua{

require("char-def.lua")

local function mirror(hlist)
    local rtl = false
    if tex.textdir == "TRT" then
        rtl = true
    elseif tex.textdir == "TLT" then
        rtl = false
    end
    for n in node.traverse(hlist) do
        if n.id == node.id('hlist') then
            mirror(n.list)
        end
        if n.id == node.id("dir") then
            if n.dir == "+TRT" or n.dir == "-TLT" then
                rtl = true
            elseif n.dir == "-TRT" or n.dir == "+TLT" then
                rtl = false
            end
        end
        if n.id == node.id('glyph') then
            if rtl then
                local char = n.char
                local mirror = characters.data[char].mirror
                if mirror then
                    n.char = mirror
                end
            end
        end
    end
end

luatexbase.add_to_callback('pre_linebreak_filter',
    function(h)
        mirror(h)
        return true
    end,
'mirror')

}
\usepackage{fontspec}
\setmainfont{Alef}
\newcommand{\Heb}[1]{\bgroup\textdir TRT\fontspec[Script=Hebrew, Language=Hebrew]{Alef}#1\egroup}
\begin{document}
    Alpha (Beta) Gamma;
    \Heb{אלף (בית) גימל};
    A (B) C
\end{document}

Another option is to use babel's data file, without the package itself. The code is pretty much the same, just few adjustments are needed:

\documentclass{article}
\directlua{

require("babel-data-bidi.lua")

local function mirror(hlist)
    local rtl = false
    if tex.textdir == "TRT" then
        rtl = true
    elseif tex.textdir == "TLT" then
        rtl = false
    end
    for n in node.traverse(hlist) do
        if n.id == node.id('hlist') then
            mirror(n.list)
        end
        if n.id == node.id("dir") then
            if n.dir == "+TRT" or n.dir == "-TLT" then
                rtl = true
            elseif n.dir == "-TRT" or n.dir == "+TLT" then
                rtl = false
            end
        end
        if n.id == node.id('glyph') then
            if rtl then
                local char = n.char
                local mirror = Babel.characters[char]
                if mirror then
                    n.char = mirror.m
                end
            end
        end
    end
end

luatexbase.add_to_callback('pre_linebreak_filter',
    function(h)
        mirror(h)
        return true
    end,
'mirror')

}
\usepackage{fontspec}
\setmainfont{Alef}
\newcommand{\Heb}[1]{\bgroup\textdir TRT\fontspec[Script=Hebrew, Language=Hebrew]{Alef}#1\egroup}
\begin{document}
    Alpha (Beta) Gamma;
    \Heb{אלף (בית) גימל};
    A (B) C
\end{document}

Both produce the following:

As a side note, it is preferable to not use standalone class for MWE's