How to produce such a matrix with braces in LaTeX? [duplicate]

Question

Currently I am working on an assignment that needs me to typeset the following Sylvester matrix in LaTeX.

I have no idea how to typeset the two braces on the right side so I would like to ask for it. I am using the document class of amsart and I can implement any package if needed.

Thanks in advance..

ps. The enclosed is what I have so far

\begin{align*}
Syl_i(A,B)=
\underbrace{
\begin{bmatrix}
a_m & \cdots & a_0 & & \\
&\ddots & \cdots & \ddots & \\
& & a_m & \cdots & a_0 \\
b_n & \cdots & b_0 & & \\
&\ddots & \cdots & \ddots & \\
& &b_n & \cdots &b_0 \\
\end{bmatrix}
}_{m+n-i}
\end{align*}

The problem is that I don't know how to add the two braces right to the matrix..

Answer 1

You can do this with some \phantom and \lefteqn commands. The idea is to stack two \phantom matrices to the right with the same entries as the upper and lower halves of the original. To get the \underbrace inside the brackets requires a matrix with no delimiters. The square brackets are added before and after with additional \phantom matrices.

If you want to add an additional column to the main matrix to line up the \ddots, everything else will adjust automatically.

Here's the code:

\documentclass{article}
\usepackage{amsmath}

\begin{document}

\[
Syl_i(A,B)=\left[\phantom{\begin{matrix}a_0\\ \ddots\\a_0\\b_0\\ \ddots\\b_0 \end{matrix}}
\right.\hspace{-1.5em}
\underbrace{\begin{matrix}
a_m & \cdots & a_0 & \\
\ddots & & \ddots & \\
 & a_m & \cdots & a_0 \\
b_n & \cdots & b_0 & \\
\ddots & & \ddots & \\
 & b_n & \cdots & b_0
\end{matrix}}_{m+n-i}
\hspace{-1.5em}
\left.\phantom{\begin{matrix}a_0\\ \ddots\\a_0\\b_0\\ \ddots\\b_0 \end{matrix}}\right]\hspace{-1em}
\begin{tabular}{l}
$\left.\lefteqn{\phantom{\begin{matrix} a_0\\ \ddots\\ a_0\ \end{matrix}}}\right\}n-i$\\
$\left.\lefteqn{\phantom{\begin{matrix} b_0\\ \ddots\\ b_0\ \end{matrix}}} \right\}m-i$
\end{tabular}
\]

\end{document}