dotted or dashed brackets in bmatrix pmatrix

Question

I'd welcome versions of the bmatrix and pmatrix environments that dash or dot the brackets. A MWE is below. I found another question that explains how to draw dashed curly braces, but I'd like an environment that behaves in every way like bmatrix or pmatrix except for the dashed pattern. How would I accomplish that? My guess is that bmatrix is much easier than pmatrix.

\documentclass{article}

\usepackage{amsmath}


\begin{document}

\begin{equation}
\begin{bmatrix}
 0 & 1 \\ 2& 3
\end{bmatrix}
\end{equation}
I want the brackets of this matrix dashed or densely dotted.
\end{document}

Answer 1

See below for expanded answer.

Here is the dBmatrix and dpmatrix environments, using code from the cited question and the environ package. My tikz is primitive, so I am still trying to figure out how to get bracket decorations.

\documentclass{article}
\usepackage{tikz,amsmath,environ}
\usetikzlibrary{decorations.pathreplacing,calligraphy}
\NewEnviron{dBmatrix}{\setbox0=\hbox{$\,\begin{matrix}\BODY\end{matrix}\,$}%
  \setbox2=\hbox{\begin{tikzpicture}
    \draw [dashed, thick, decorate, decoration={brace, amplitude=5pt}] (0,\botdim)--(0,\topdim);
    \copy0
    \draw [dashed, thick, decorate, decoration={brace, amplitude=5pt}] (0,\topdim)--(0,\botdim);
  \end{tikzpicture}}
  \vcenter{\hbox{\copy2}}}
\NewEnviron{dpmatrix}{\setbox0=\hbox{$\,\begin{matrix}\BODY\end{matrix}\,$}%
  \setbox2=\hbox{\begin{tikzpicture}
    \draw [dashed, thick, decorate, decoration={calligraphic straight parenthesis, amplitude=1pt}] (0,\botdim)--(0,\topdim);
    \copy0
    \draw [dashed, thick, decorate, decoration={calligraphic straight parenthesis, amplitude=1pt}] (0,\topdim)--(0,\botdim);
  \end{tikzpicture}}
  \vcenter{\hbox{\copy2}}}
\def\topdim{\dimexpr+\ht0+.5\ht\strutbox-.5\dp\strutbox-3pt\relax}
\def\botdim{\dimexpr-\ht0+.5\ht\strutbox-.5\dp\strutbox+3pt\relax}
\begin{document}
\[
\begin{Bmatrix}
   0 & 1 \\ 2& 3
\end{Bmatrix}%
\quad
\begin{dBmatrix}
   0 & 1 \\ 2& 3
\end{dBmatrix}
\]
\[
\begin{pmatrix}
   0 & 1 \\ 2& 3\\4 & 5
\end{pmatrix}%
\quad
\begin{dpmatrix}
   0 & 1 \\ 2& 3\\4& 5
\end{dpmatrix}
\]
\end{document}

---

Employing Peter Grill's answer at draw round/rectangular bracket embracing nodes in tikz allowed for the definition of rounded parens as well as square bracket decorations. Thus, the look here is slightly different than above. So here, dpmatrix is redefined from above and dbmatrix is introduced.

\documentclass{article}
\usepackage{tikz,amsmath,environ}
\usetikzlibrary{decorations.pathreplacing,calligraphy,calc}
\tikzset{
    ncbar angle/.initial=90,
    ncbar/.style={
        to path=(\tikztostart)
        -- ($(\tikztostart)!#1!\pgfkeysvalueof{/tikz/ncbar angle}:(\tikztotarget)$)
        -- ($(\tikztotarget)!($(\tikztostart)!#1!\pgfkeysvalueof{/tikz/ncbar angle}:(\tikztotarget)$)!\pgfkeysvalueof{/tikz/ncbar angle}:(\tikztostart)$)
        -- (\tikztotarget)
    },
    ncbar/.default=0.5cm,
}

\tikzset{square left brace/.style={ncbar=0.5ex}}
\tikzset{square right brace/.style={ncbar=-0.5ex}}

\tikzset{round left paren/.style={ncbar=0.3cm,out=115,in=-115}}
\tikzset{round right paren/.style={ncbar=0.3cm,out=65,in=-65}}
%
\NewEnviron{dBmatrix}{\setbox0=\hbox{$\,\begin{matrix}\BODY\end{matrix}\,$}%
  \setbox2=\hbox{\begin{tikzpicture}
    \draw [dashed, thick, decorate, decoration={brace, amplitude=5pt}] (0,\botdim)--(0,\topdim);
    \copy0
    \draw [dashed, thick, decorate, decoration={brace, amplitude=5pt}] (0,\topdim)--(0,\botdim);
  \end{tikzpicture}}
  \vcenter{\hbox{\copy2}}%
}
\NewEnviron{dpmatrix}{\setbox0=\hbox{$\,\begin{matrix}\BODY\end{matrix}\,$}%
  \setbox2=\hbox{\begin{tikzpicture}
    \draw [dashed, thick] (0,\botdim) to [round left paren] (0,\topdim);
    \copy0
    \draw [dashed, thick] (0,\botdim) to [round right paren] (0,\topdim);
  \end{tikzpicture}}
  \vcenter{\hbox{\copy2}}%
}
\NewEnviron{dbmatrix}{\setbox0=\hbox{$\,\begin{matrix}\BODY\end{matrix}\,$}%
  \setbox2=\hbox{\begin{tikzpicture}
    \draw [dashed, thick] (0,\botdim) to [square left brace] (0,\topdim);
    \copy0
    \draw [dashed, thick] (0,\botdim) to [square right brace] (0,\topdim);
  \end{tikzpicture}}
  \vcenter{\hbox{\copy2}}%
}
\def\topdim{\the\dimexpr+\ht0+.5\ht\strutbox-.5\dp\strutbox-3pt\relax}
\def\botdim{\the\dimexpr-\ht0+.5\ht\strutbox-.5\dp\strutbox+3pt\relax}
\begin{document}
\[
\begin{Bmatrix}
   0 & 1 \\ 2& 3
\end{Bmatrix}%
\quad
\begin{dBmatrix}
   0 & 1 \\ 2& 3
\end{dBmatrix}
\]
\[
\begin{pmatrix}
   0 & 1 \\ 2& 3\\4 & 5
\end{pmatrix}%
\quad
\begin{dpmatrix}
   0 & 1 \\ 2& 3\\4& 5
\end{dpmatrix}
\]
\[
\begin{bmatrix}
   0 & 1 \\ 2& 3
\end{bmatrix}%
\quad
\begin{dbmatrix}
   0 & 1 \\ 2& 3
\end{dbmatrix}
\]
\end{document}