MathJax is off-topic on this site, but in LaTeX you can use kbordermatrix
. Download the kbordermatrix.sty
file from the provided link and place it in the same directory as your document. Then you can use:
\documentclass{article}
\usepackage{amsmath}
\usepackage{kbordermatrix}
\begin{document}
\[
\newcommand\col[1]{\parbox{1cm}{\centering\scriptsize#1}}
\omega(\mu\nu|\theta) =
\kbordermatrix{
& & \col{$\mu$th column} & & \col{$\nu$th column} & \\
& & \vdots & & \vdots & \\
\text{$\mu$th row} & \cdots & \cos\theta & \cdots & \sin\theta & \cdots \\
& & \vdots & & \vdots & \\
\text{$\nu$th row} & \cdots & -\sin\theta & \cdots & \cos\theta & \cdots \\
& & \vdots & & \vdots & \\
}
\]
\end{document}
Or using blkarray
for more flexibility:
\documentclass{article}
\usepackage{amsmath}
\usepackage{blkarray}
\begin{document}
\[
\newcommand\col[1]{\parbox{1cm}{\centering\scriptsize#1}}
\newcommand\row[1]{\text{\scriptsize#1}}
\omega(\mu\nu|\theta) =
\begin{blockarray}{cccccc}
& \col{$\mu$th column} & & \col{$\nu$th column} & & \\
\begin{block}{[ccccc]c}
& \vdots & & \vdots & & \\
\cdots & \cos\theta & \cdots & \sin\theta & \cdots & \row{$\mu$th row} \\
& \vdots & & \vdots & & \\
\cdots & -\sin\theta & \cdots & \cos\theta & \cdots & \row{$\nu$th row} \\
& \vdots & & \vdots & & \\
\end{block}
\end{blockarray}
\]
\end{document}
In MathJax you can screw around with some manual positioning to piece together something that looks sort of right. Of course this will break when you change the font or the font size.
$$
\omega(\mu\nu|\theta) =
\left[
\matrix{
& \vdots & & \vdots & \\
\cdots & \cos\theta & \cdots & \sin\theta & \cdots \\
& \vdots & & \vdots & \\
\cdots & -\sin\theta & \cdots & \cos\theta & \cdots \\
& \vdots & & \vdots & \\
}
\right]
\matrix{
\phantom{\vdots} \\
\text{$\mu$th row} \\
\phantom{\vdots} \\
\text{$\nu$th row} \\
\phantom{\vdots} \\
}
\hskip-14.3em\raise 13ex \hbox{$\mu$th}\hskip-2.5em\raise 10ex \hbox{column}
\hskip 3.5em\raise 13ex \hbox{$\nu$th}\hskip-2.5em\raise 10ex \hbox{column}
$$
Live example
kbordermatrix
.