Parentheses differ (XeLaTeX, fontspec, newtxmath, libertine)

Question

After Internal error: bad native font flag I stumbled upon another weird problem with the combination XeLaTeX/fontspec/Libertine/newtxmath:

\documentclass{scrartcl}
\usepackage[no-math]{fontspec}
\usepackage{libertineotf}
\usepackage{amsmath}
\usepackage[libertine]{newtxmath}

\begin{document}
    \begin{equation}
              ( \frac{1}{1}       )
         \left( \frac{1}{2} \right)
         \bigl( \frac{1}{3}  \bigr)
         \Bigl( \frac{1}{4}  \Bigr)
        \biggl( \frac{1}{5} \biggr)
        \Biggl( \frac{1}{6} \Biggr)
    \end{equation}
\end{document}

The up-scaled result is:
http://img225.imageshack.us/img225/5576/bracketsv.png

It seems to me that the Libertine font is used for ( and and CM for ).
Compare the closing parentheses from the above image to the following where the fontspec package was not used:
http://img94.imageshack.us/img94/2015/brackets2.png
The parentheses have the correct size but again CM is used for the numbers.
pdfLaTeX and the libertine package produce the same output as XeLaTeX without fontspec.

Answer 1

This seems to depend on Linux Libertine, that gets in the way with \left( (which is implicitly used also in the \big commands), so that \left( chooses the parenthesis from Libertine and not from the newtxmath defined font.

Other fonts instead of Linux Libertine don't exhibit this issue.

A workaround. Add the following lines after the definition of the fonts:

\DeclareSymbolFont{parenthesis}{OT1}{ntxr}{m}{n}
\DeclareMathDelimiter{(}{\mathopen}{parenthesis}{"28}{largesymbols}{"00}
\DeclareMathDelimiter{)}{\mathclose}{parenthesis}{"29}{largesymbols}{"01}

We are explicitly requesting that the parentheses are chosen from the newtxmath font. This gives another issue, though: the parentheses in math mode (normal size) will be different from the text mode ones.

Answer 2

This seems to be bug in the libertine fonts. When you use newtxmath it will set the operator font (the one used for small parentheses) to the roman font (libertine). For some reason the font will use the operator font for large opening parentheses too instead of the correct large symbols font. The handling of closing parentheses is correct The problem disappears when you don't use the no-math option of fontspec as in this case the operator font is reset to cmr.

The bug is present in version 5.1.3 and also in the newest 5.3 of the fonts, with 4.7.5. it works fine. I boiled down the example to this (if exhibits the problem with xelatex + lualatex for me):

\documentclass{scrartcl}
\usepackage[no-math]{fontspec}
%\setmainfont{LinLibertine_Re-4.7.5.otf} %works
\setmainfont{LinLibertine_R.otf} % fails, version 5.1.3 and 5.3
\DeclareSymbolFont{operators}{\encodingdefault}{\rmdefault}{m}{n}
\pagestyle{empty}

% without no-math the following is used:
% \DeclareSymbolFont{legacymaths}{OT1}{cmr}{m}{n}
% \SetSymbolFont{legacymaths}{bold}{OT1}{cmr}{bx}{n}
% \DeclareMathDelimiter{(}{\mathopen} {legacymaths}{40}{largesymbols}{0}
% \DeclareMathDelimiter{)}{\mathclose}{legacymaths}{41}{largesymbols}{1}

\DeclareMathDelimiter{(}{\mathopen} {operators}{40}{largesymbols}{0}
\DeclareMathDelimiter{)}{\mathclose}{operators}{41}{largesymbols}{1}
\begin{document}
$        \Bigg( \Biggl( % libert  %without no-math: txex
         \Bigg) \Biggr) % txex    %without no-math: txex
$
\end{document}

I think you should make a bug report to the font authors.

Btw: I get a warning from xdvipdfmx which is perhaps related:

** WARNING ** agl: fix parenrightBigg --> parenright.Bigg