Whilst I agree with the comments that this is most likely due to the viewer, it is nonetheless irritating. I believe that when printed this effect is not seen, but since TeX is used quite a lot nowadays for presentations then how it looks in a particular viewer is becoming more important than it was. So despite it likely to be an issue with the viewer, it is still an issue worth solving.
The problem is that TeX is building the delimiters out of pieces and the viewer is rendering these pieces in a non-optimal manner and showing the spaces between them. (Unfortunately, I also dislike it when it does the opposite and overlaps the pieces as I can see the overlaps as extra thick pieces!).
A simple way to solve it is to use a font that has proper scalable delimiters, as mentioned in the accepted answer to Vertical vectors in angle brackets.
Another way of solving it would be to draw the delimiters as a single entity, rather than build them piecemeal. We had a question about replacing the AMS matrices with TikZ matrices: Obstacles to simulating an amsmath matrix by a TiKZ matrix of math nodes. In answering that question, I figured out a drop-in replacement for the standard AMS matrix environments using TikZ matrices instead. However, for the delimiters then it uses the standard delimiters so doesn't solve this problem.
But it wasn't too difficult to modify the code there to make it so that it draws the delimiters using TikZ code. By using my calligraphy package, I could even get nice tapering on the brace and parenthesis versions. At time of writing, the calligraphy package is not on CTAN, but you can get it from http://bazaar.launchpad.net/~tex-sx/tex-sx/development/files (you also need the spath.dtx file).
I'll put the code in a moment, first I want to show the result. The following is a matrix of matrices. Each row is a particular type (bmatrix, pmatrix, etc). The first in each row has TikZ draw the delimiters. The second has TikZ use "standard" delimiters. The third is the corresponding matrix command from the AMS packages.
The code follows.
\documentclass{article}
%\url{https://tex.stackexchange.com/q/26866/86}
\usepackage{amsmath,tikz}
\usetikzlibrary{matrix,decorations}
\usepackage{calligraphy}
\newlength\mtxrowsep
\setlength\mtxrowsep{1.5ex}
\newlength\mtxcolsep
\setlength\mtxcolsep{2\arraycolsep}
\makeatletter
\pgfdeclaredecoration{doubled lineto}{brace}
{
\state{brace}[width=+\pgfdecoratedremainingdistance,next state=final]
{
\pgfsyssoftpath@setcurrentpath{\pgfutil@empty}
\pgfpathmoveto{\pgfpointorigin}
\pgfpathlineto{\pgfqpoint{\pgfdecoratedremainingdistance}{0pt}}
\pgfpathmoveto{\pgfqpoint{0pt}{\pgfdecorationsegmentamplitude}}
\pgfpathlineto{\pgfqpoint{\pgfdecoratedremainingdistance}{\pgfdecorationsegmentamplitude}}
}
\state{final}{}%
}
\pgfdeclaredecoration{bracket}{brace}
{
\state{brace}[width=+\pgfdecoratedremainingdistance,next state=final]
{
\pgfsyssoftpath@setcurrentpath{\pgfutil@empty}
\pgfpathmoveto{\pgfpointorigin}
\pgfpathlineto{\pgfqpoint{0pt}{\pgfdecorationsegmentamplitude}}
\pgfpathlineto{\pgfqpoint{\pgfdecoratedremainingdistance}{\pgfdecorationsegmentamplitude}}
\pgfpathlineto{\pgfqpoint{\pgfdecoratedremainingdistance}{0pt}}
}
\state{final}{}%
}
\makeatother
\expandafter\def\csname delimiter \string\lbrace\endcsname{calligraphic brace}
\expandafter\def\csname delimiter \string(\endcsname{calligraphic curved parenthesis}
\expandafter\def\csname delimiter \string|\endcsname{lineto}
\expandafter\def\csname delimiter \string\|\endcsname{doubled lineto}
\expandafter\def\csname delimiter \string[\endcsname{bracket}
\expandafter\def\csname delimiter \string\rbrace\endcsname{calligraphic brace}
\expandafter\def\csname delimiter \string)\endcsname{calligraphic curved parenthesis}
\expandafter\def\csname delimiter \string]\endcsname{bracket}
\tikzset{
ams/.style={
baseline=-.7ex,
every delimiter/.style={yshift=-1pt},
every left delimiter/.style={xshift=2pt},
every right delimiter/.style={xshift=-2pt},
every node/.style={inner sep=0pt},
execute at end picture={
\path (current bounding box.east) ++(\pgfkeysvalueof{/tikz/ams matrix xsep},0) (current bounding box.west) ++(-\pgfkeysvalueof{/tikz/ams matrix xsep},0);
},
},
ams matrix xsep/.initial={.5ex},
ams matrix/.style={
inner sep=1pt,
column sep=\mtxcolsep,
row sep=\mtxrowsep,
% ampersand replacement=\&,
matrix of math nodes,
},
ams delimiters/.style args={#1,#2}{
left delimiter={#1},
right delimiter={#2},
},
delimiters/.style args={#1,#2}{
\pgfkeysvalueof{/tikz/matrix delimiter type} delimiters={{#1},{#2}},
},
matrix delimiter type/.initial={tikz},
tikz delimiter style/.style={
thick,
},
tikz delimiters/.style args={#1,#2}{
left tikz delimiter={#1},
right tikz delimiter={#2},
},
left tikz delimiter/.style={
render left tikz delimiter/.expand once={\csname delimiter \string#1\endcsname}
},
render left tikz delimiter/.style={
append after command={(\tikzlastnode.north west) edge[tikz delimiter style,decorate,decoration={mirror,#1}] (\tikzlastnode.south west)}
},
right tikz delimiter/.style={
render right tikz delimiter/.expand once={\csname delimiter \string#1\endcsname}
},
render right tikz delimiter/.style={
append after command={(\tikzlastnode.north east) edge[tikz delimiter style,decorate,decoration={#1}] (\tikzlastnode.south east)}
},
bmatrix/.style={
ams,
every matrix/.style={
ams matrix,
delimiters={[,]},
}
},
Bmatrix/.style={
ams,
every matrix/.style={
ams matrix,
delimiters={\lbrace,\rbrace},
}
},
pmatrix/.style={
ams,
every matrix/.style={
ams matrix,
delimiters={(,)},
}
},
vmatrix/.style={
ams,
every matrix/.style={
ams matrix,
delimiters={|,|},
}
},
Vmatrix/.style={
ams,
every matrix/.style={
ams matrix,
delimiters={\|,\|},
}
},
}
\let\matamp=&
\catcode`\&=13
\makeatletter
\def&{\iftikz@is@matrix
\pgfmatrixnextcell
\else
\matamp
\fi}
\makeatother
%\usepackage{environ}
\def\endtikzmatrix{\\\egroup;\end{tikzpicture}}
\foreach \mtype in {b,B,p,v,V} {
\expandafter\xdef\csname tikz\mtype matrix\endcsname{%
\noexpand\begin{tikzpicture}[\mtype matrix,matrix delimiter type={tikz}]
\noexpand\matrix \noexpand\bgroup}
\expandafter\xdef\csname ams\mtype matrix\endcsname{%
\noexpand\begin{tikzpicture}[\mtype matrix,matrix delimiter type={ams}]
\noexpand\matrix \noexpand\bgroup}
\expandafter\global\expandafter\let\csname endtikz\mtype matrix\endcsname=\endtikzmatrix
\expandafter\global\expandafter\let\csname endams\mtype matrix\endcsname=\endtikzmatrix
}
\begin{document}
\foreach \mtype in {b,B,p,v,V} {
\edef\metype{\mtype matrix}
\edef\tmetype{tikz\mtype matrix}
\edef\ametype{ams\mtype matrix}
\begin{gather*}
\begin{\tmetype} a & b \\ c & d \end{\tmetype}
\begin{\ametype} a & b \\ c & d \end{\ametype}
\begin{\metype} a & b \\ c & d \end{\metype}
\end{gather*}
}
\end{document}
The change from the code as in Obstacles to simulating an amsmath matrix by a TiKZ matrix of math nodes is that after rendering the matrix, we draw a line down each side. But we decorate the path according to the type of delimiter chosen. If we didn't use the calligraphy package, we'd have to change some of the choices of decorations in the list near the beginning:
\expandafter\def\csname delimiter \string\lbrace\endcsname{calligraphic brace}
\expandafter\def\csname delimiter \string(\endcsname{calligraphic curved parenthesis}
\expandafter\def\csname delimiter \string\rbrace\endcsname{calligraphic brace}
\expandafter\def\csname delimiter \string)\endcsname{calligraphic curved parenthesis}
calligraphic brace could be replaced by brace and calligraphic curved parenthesis by bent.
One advantage of this approach is that it allows for more variation in the delimiters. We can define new decorations, and can style the paths in a variety of different ways.
[]surrounding the matrix? For one, this may just be an artifact of the PDF viewer you're using. For example, at 200% zoom in Adobe Acrobat 9, there is no difference in the line thickness and 'gaps'. In fact, when I look closely at your figures, I see similar (albeit less noticeable, but still there) 'white separations' in[].setspacepackage, and then use\onehalfspacing.