The package cmbright
doesn't make sense with XeLaTeX/fontspec
, because it loads legacy fonts.
You can use the CM-Unicode version of CMBright.
\documentclass[10pt,a4paper]{article}
\usepackage{fontspec}
\usepackage{polyglossia}
\setmainlanguage{french}
\setmainfont{cmunb}[
Extension=.otf,
UprightFont=*mr,
ItalicFont=*mo,
BoldFont=*bx,
BoldItalicFont=*xo,
]
\author{Plop}
\title{Test fonts xelatex}
\usepackage{lipsum}
\begin{document}
\maketitle
Abc \textbf{Abc} \textit{Abc} \textbf{\textit{Abc}}
\lipsum[1-4]
\end{document}

You may want to use semibold, in this case change the declaration into
\setmainfont{cmunb}[
Extension=.otf,
UprightFont=*mr,
ItalicFont=*mo,
BoldFont=*so, % semibold
BoldItalicFont=*sr, % semibold oblique
]

If the CM-Unicode fonts are installed also as system fonts, you can declare them more easily by name
\setmainfont{CMU Bright}
that would choose the semibold variant. For getting the bold variant use
\setmainfont{CMU Bright}[
BoldFont=* Bold Extended,
BoldItalicFont=* Bold Extended Oblique,
]