With nicematrix
:
\documentclass{article}
\usepackage{nicematrix}
\begin{document}
$\begin{pNiceMatrix}[first-row,first-col,code-for-first-row=\rotate]
& |n_1l_1m_1\rangle& |n_1l_1m_1\rangle& |n_1l_1m_1\rangle\\
|n_1l_1m_1\rangle & 0.8 & 0.2 & 0 \\
|n_1l_1m_1\rangle& 0.3 & 0.4 & 0.3 \\
|n_1l_1m_1\rangle& 0.2 & 0.1 & 0.7
\end{pNiceMatrix}$
\end{document}

By default, Overleaf uses an old version of nicematrix
. However, you can provide it the newest version of nicematrix.sty
. Go here. At the top of the page, you have the latest version of nicematrix
. It's the revision 56256. You do a right click on download
and you save the file under the name nicematrix.sty
. Put that file next to your file in your Overleaf project (buttom import
next to new file
).
\rotatebox{80}{$\ket{n_1l_1m_1}$}