An almost direct modification of \bordermatrix:
\documentclass{article}
\makeatletter
\def\bbordermatrix#1{\begingroup \m@th
\@tempdima 4.75\p@
\setbox\z@\vbox{%
\def\cr{\crcr\noalign{\kern2\p@\global\let\cr\endline}}%
\ialign{$##$\hfil\kern2\p@\kern\@tempdima&\thinspace\hfil$##$\hfil
&&\quad\hfil$##$\hfil\crcr
\omit\strut\hfil\crcr\noalign{\kern-\baselineskip}%
#1\crcr\omit\strut\cr}}%
\setbox\tw@\vbox{\unvcopy\z@\global\setbox\@ne\lastbox}%
\setbox\tw@\hbox{\unhbox\@ne\unskip\global\setbox\@ne\lastbox}%
\setbox\tw@\hbox{$\kern\wd\@ne\kern-\@tempdima\left[\kern-\wd\@ne
\global\setbox\@ne\vbox{\box\@ne\kern2\p@}%
\vcenter{\kern-\ht\@ne\unvbox\z@\kern-\baselineskip}\,\right]$}%
\null\;\vbox{\kern\ht\@ne\box\tw@}\endgroup}
\makeatother
\begin{document}
$
M = \bbordermatrix{~ & x & y \cr
A & 1 & 0 \cr
B & 0 & 1 \cr}
$
\end{document}
I acted on \left( and \right), but also changed the value given to \@tempdima, smaller because brackets are slimmer than parentheses.
If you don't want such a messy code, just do
\usepackage{etoolbox}
\let\bbordermatrix\bordermatrix
\patchcmd{\bbordermatrix}{8.75}{4.75}{}{}
\patchcmd{\bbordermatrix}{\left(}{\left[}{}{}
\patchcmd{\bbordermatrix}{\right)}{\right]}{}{}
that does exactly the same changes.

kbordermatrix(look at the last item for documentation) environment available for such purposes. – percusse May 9 '12 at 12:20\matrixcommand? – Werner May 9 '12 at 15:52