# Horizontal curly braces inside matrix

I'm trying to make this determinant:

But I don't know how to put the curly braces. I've tried with \underbrace but I get the brace only under a_m, not the entire space.

My try:

\documentclass{article}
\usepackage{amsmath}

\begin{document}

$$R(f,g)=\begin{vmatrix} a_0 & & & & b_0 & & & \\ a_1 & a_0 & & & b_1 & b_0 & & \\ a_2 & a_1 & a_0 & & b_2 & b_1 & b_0 & \\ & \cdots & & & & \cdots & & \\ & & & a_m & & & & b_n \\ \end{vmatrix}$$

\end{document}

Here there is my idea: to have the curly braces inside the matrix of your determinant I have used always underbrace command.

\documentclass[a4paper,12pt]{article}
\usepackage{amsmath,amssymb}
\begin{document}
$$R(f,g)= \begin{vmatrix} \underbrace{\begin{matrix} a_0 & & & \\ a_1 & a_0 & & \\ a_2 & a_1 & a_0 & \\ & \ldots & & \\ & & & a_m \end{matrix}}_{n \text{ columns}} & \underbrace{\begin{matrix} b_0 & & & \\ b_1 & b_0 & & \\ b_2 & b_1 & b_0 & \\ & \ldots & & \\ & & & b_m \end{matrix}}_{m \text{ columns}}\\ \end{vmatrix}$$
\end{document}

This MWE have the curly brackets more inside:

\documentclass[a4paper,12pt]{article}
\usepackage{amsmath,amssymb}
\begin{document}
$$R(f,g)=\begin{array}{c} \begin{vmatrix} \underbrace{\begin{matrix} a_0 & & & \\ a_1 & a_0 & & \\ a_2 & a_1 & a_0 & \\ & \ldots & & \\ & & & a_m \end{matrix}}_{n \text{ columns}} & \underbrace{\begin{matrix} b_0 & & & \\ b_1 & b_0 & & \\ b_2 & b_1 & b_0 & \\ & \ldots & & \\ & & & b_m \end{matrix}}_{m \text{ columns}}\\ & \end{vmatrix} \end{array}$$
\end{document}

You can use nicematrix. Some work is needed to pull the braces nearer the matrix, but not so much: the idea is to use a phantom matrix to set the width for the underbrace; then it is raised up.

\documentclass{article}
\usepackage{amsmath}
\usepackage{nicematrix}

\usepackage{lipsum} % mock text to show spacing around the display

\begin{document}

\lipsum[1]
$$R(f,g)= \begin{vNiceMatrix}[last-row] a_0 & & & & b_0 & & & \\ a_1 & a_0 & & & b_1 & b_0 & & \\ a_2 & a_1 & a_0 & & b_2 & b_1 & b_0 & \\ \multicolumn{3}{c}{\cdots} & & \multicolumn{3}{c}{\cdots} & \\ & & & a_m & & & & b_n \\ \multicolumn{4}{c}{% \raisebox{1ex}{ \underbrace{% \hphantom{\begin{matrix} a_0 & a_1 & a_2 & a_m\end{matrix}} }_{\text{n columns}} }% } & \multicolumn{4}{c}{% \raisebox{1ex}{ \underbrace{% \hphantom{\begin{matrix} b_0 & b_1 & b_2 & b_n\end{matrix}} }_{\text{m columns}} }% } \end{vNiceMatrix}$$
\lipsum[2]

\end{document}

Note the \multicolumn instructions in the body to avoid misalignments.

I would place the two underbraced explanatory expressions outside the vertical bars. This may be done by with the help of 2 "\smashed" matrix environments and two judiciously chosen typographic struts -- one inside the \left\vert ... \right\vert group, containing a matrix with 5 columns, and one outside that group, containing a matrix with 7 columns (and a \smash[t] directive).

\documentclass{article}
\usepackage{amsmath}
\begin{document}
$$R(f,g)= \left\vert \smash{\underbrace{\begin{matrix} a_0 & & & \\ a_1 & a_0 & & \\ a_2 & a_1 & a_0 & \\ & \cdots \\ & & & a_m \end{matrix}}_{\text{n columns}}}\; \smash{\underbrace{\begin{matrix} b_0 & & & \\ b_1 & b_0 & & \\ b_2 & b_1 & b_0 & \\ & \cdots \\ & & & b_n \end{matrix}}_{\text{m columns}}} %% insert a first typographic strut: \vphantom{\begin{matrix} 1\\2\\3\\4\\5 \end{matrix}} \right\vert %% insert a second typographic strut: \smash[t]{\vphantom{\begin{matrix} 1\\2\\3\\4\\5\\6\\7 \end{matrix}}}$$
\end{document}
• There is a problem with the colours of the your code. All the characters have the same gray color. Oct 18, 2020 at 20:39
• @Sebastiano - Not sure if the absence of color is a problem, but it is deliberate; see tex.meta.stackexchange.com/q/8668/5001 for more information.
– Mico
Oct 18, 2020 at 20:42
• In my humble opinion there are many things still to be fixed on TeX.SE. Oct 18, 2020 at 20:44