You could just substitute the a, b, c for the matrix or whatever you want in brackets (I've added the - before so that the label is aligned in the middle of the arrow, not the middle of the row/column). Here's an example:
\begin{equation*}
\xymatrix @R=2cm @C=2cm {
{\bullet}^{V_1} \ar[r]^-{a = \begin{pmatrix}
a_1 & a_2 \\ a_3 & a_4
\end{pmatrix}} & {\bullet}^{V_2} \ar[d]^-{b = \begin{pmatrix}
b_1 & b_2 \\ b_3 & b_4
\end{pmatrix}} \\
& {\bullet}^{V_3} \ar[ul]^-{c = \begin{pmatrix}
c_1 & c_2 \\ c_3 & c_4
\end{pmatrix}}
}
\end{equation*}
And may be you would like to write into an smallmatrix, which could be enough for your needs:
\begin{equation*}
\xymatrix @R=2cm @C=2cm {
{\bullet}^{V_1} \ar[r]^-{a = \left(\begin{smallmatrix}
a_1 & a_2 \\ a_3 & a_4
\end{smallmatrix}\right)} & {\bullet}^{V_2} \ar[d]^-{b = \left(\begin{smallmatrix}
b_1 & b_2 \\ b_3 & b_4
\end{smallmatrix}\right)} \\
& {\bullet}^{V_3} \ar[ul]^-{c = \left(\begin{smallmatrix}
c_1 & c_2 \\ c_3 & c_4
\end{smallmatrix}\right)}
}
\end{equation*}
The @R=X and the @C=Y in the xymatrix command gives you the chance to change row and column separation.
There is a package tikz-cd (here is the manual at CTAN) which, in my opinion, gives better quality and the language used is easier to understand (i.e. \arrow[hookrightarrow]{rr}{a} instead of the xymatrix \ar@{^{(}->}[rr]^-{a}; or \rar{a} instead of \ar[r]^-{a}). But, in this concrete case there is a problem: tikz-cd causes problems if you want to label an arrow with a matrix (here is a better explanation). That's why I don't include an example here.