# Everything after the & symbol in each line for the rcases environment isn't in math mode, I don't know why

\documentclass[12pt]{article}
\usepackage{mathtools}
\begin{document}

$$\begin{rcases*} \left( \nabla^2+k^2\right) & u(\mathbf{r}) =0, \quad r > a \\ & u(\mathbf{r}) =0, \quad r=a. \end{rcases*}$$

\end{document}


If you want the material after the & alignment points to be treated as being in math mode, use the rcases environment, not the rcases* environment:

\documentclass[12pt]{article}
\usepackage{mathtools}
\begin{document}
$$\begin{rcases} \left( \nabla^2+k^2\right) & u(\mathbf{r}) =0, \quad r > a \\ & u(\mathbf{r}) =0, \quad r=a. \end{rcases}$$
\end{document}


I'm not sure about rcases, but do you want the following?

\documentclass[12pt]{article}
\usepackage{mathtools}
\begin{document}

$$\left( \nabla^2+k^2\right) \begin{cases} u(\mathbf{r}) =0, & \quad r > a\\ u(\mathbf{r}) =0, & \quad r=a \end{cases}$$

\end{document}