The operator -
has greater depth than the surrounding numerals. As such, the second smallmatrix
is "taller" than the first, causing the delimiters to stretch further. The following highlights this somewhat:

\documentclass{article}
\usepackage{mathtools}% http://ctan.org/pkg/mathtools
\setlength{\fboxsep}{-\fboxrule}
\newcommand{\boxit}[1]{\text{\fbox{$#1$}}}
\begin{document}
\[
\left(\begin{smallmatrix*}[r]
\boxit{1} & \boxit{2} \\
\boxit{5} & \boxit{7}
\end{smallmatrix*}\right) \quad {\def\boxit#1{#1}
\left(\begin{smallmatrix*}[r]
\boxit{1} & \boxit{2} \\
\boxit{5} & \boxit{7}
\end{smallmatrix*}\right)} \qquad
\left(\begin{smallmatrix*}[r]
\boxit{1} & \boxit{-1} \\
\boxit{2} & \boxit{3}
\end{smallmatrix*}\right) \quad {\def\boxit#1{#1}
\left(\begin{smallmatrix*}[r]
\boxit{1} & \boxit{-1} \\
\boxit{2} & \boxit{3}
\end{smallmatrix*}\right)}
\]
\end{document}
\boxit
puts a bounding box around the element. Note how the elements in the first row of the second smallmatrix
are not vertically aligned.
The marginal increase in height is enough to increase the delimiters. This behaviour can be adjusted via setting elements like \delimitershortfall
and/or \delimiterfactor
. In the following example, I've set \delimitershortfall
to 0pt
:

\documentclass{article}
\usepackage{mathtools}% http://ctan.org/pkg/mathtools
\setlength{\fboxsep}{-\fboxrule}
\newcommand{\boxit}[1]{\text{\fbox{$#1$}}}
\begin{document}
\setlength{\delimitershortfall}{0pt}
\[
\left(\begin{smallmatrix*}[r]
\boxit{1} & \boxit{2} \\
\boxit{5} & \boxit{7}
\end{smallmatrix*}\right) \quad {\def\boxit#1{#1}
\left(\begin{smallmatrix*}[r]
\boxit{1} & \boxit{2} \\
\boxit{5} & \boxit{7}
\end{smallmatrix*}\right)} \qquad
\left(\begin{smallmatrix*}[r]
\boxit{1} & \boxit{-1} \\
\boxit{2} & \boxit{3}
\end{smallmatrix*}\right) \quad {\def\boxit#1{#1}
\left(\begin{smallmatrix*}[r]
\boxit{1} & \boxit{-1} \\
\boxit{2} & \boxit{3}
\end{smallmatrix*}\right)}
\]
\end{document}
Read some more about these settings in Automatic size adjustment for nested parentheses.
\vphantom
seems to give the desired size:\item $A = \left( \begin{smallmatrix*}[r] 1 & \vphantom{-}2 \\ 5 & 7 \end{smallmatrix*} \right)$
– Gonzalo Medina Mar 14 '13 at 4:27mathtools
but (as someone else has said) the fact that the minus has extra depth. this is built into the font (at least for computer modern), the reasoning being that plus and minus should have the same dimensions and spacing. – barbara beeton Mar 14 '13 at 16:51