Would it be possible to use Libertine otf font (using LuaLaTeX and, for example, package libertine for automatic settings) for everything but in math mode where I'd like to use newtxmath and its libertine option (which produces acceptable output with pdfLaTeX)?
Neither :
\usepackage[libertine]{newtxmath}
\usepackage[no-math]{fontspec}
\usepackage{libertine}
nor
\usepackage[libertine]{newtxmath}
\usepackage[no-math]{fontspec}
\setmainfont[Ligatures={TeX}]{Linux Libertine O}
are successful. I get CM font for maths using the following:
%!TEX program = lualatex
\documentclass[]{scrbook}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage[libertine]{newtxmath}
\usepackage[no-math]{fontspec}
\usepackage{libertine}
\begin{document}
\begin{equation}
\mathit{TRUC} = \frac{1}{10} \times \sum _{d=0.2}^{12.8} \frac{\pi \times d \times A_{d}}{E_{d}}
\end{equation}
\end{document}
xelatex
andlualatex
. Please add a complete minimal working example (MWE) which exhibits the error you are experiencing.