How to get nicematrix diagonal dots in the right place

Question

I'm trying to write a matrix with dots in it using the nicematrix package. I'm struggling to get the dots in the right place. My current best attempt looks like this:

The lower triangle of the matrix consists of zeros, which are nicely connected by dots to make a triangle. The diagonal is filled with the term CA^{-1}B. I need those diagonal dots to connect nicely, as they do with the zeros. I think the current diagonal dots that I have marked are aimed towards the rule above the target rule, but I'm not sure of this. An MWE of my best attempt is:

\documentclass{article}
\usepackage{amsmath,nicematrix}

\begin{document}
$\begin{bNiceMatrix}[nullify-dots] CA^{-1}B & CA^{-2}B & \Cdots & CA^{-p}B & CA^{-(p+1)}B \\
0 & CA^{-1}B & \Cdots & CA^{-(p-1)}B & CA^{-p}B \\
\Vdots & \Ddots & \Ddots & \Vdots & \Vdots \\
& & & CA^{-1}B & CA^{-2}B \\
0 & \Cdots & & 0 & CA^{-1}B \end{bNiceMatrix}$
\end{document}

How do I make the diagonal dots behave nicely?

Answer 1

What you are seeing is a consequence of the fact that nicematrix draws lines of diagonal dots parallel to each other. The first such line drawn determines the direction (see Technical remarks, Diagonal lines in the manual). Using \extrarowheight, things like \strut in matrix elements and \noalign{\kern -2pt} between lines, one can influence vertical positioning of rows, and thus how lines with diagonal dots connect to matrix elements. Another possibility is to draw lines with diagonal dots in a different order so as to use another direction for them.

Here is my proposal. nicematrix v3.13 or later is required for the xdots/shorten option, but if your nicematrix version is 3.10, 3.11 or 3.12, you can just replace xdots/shorten with dotted-lines-margin.

\documentclass{article}
\usepackage{amsmath}
\usepackage{nicematrix}

\begin{document}

$\setlength{\extrarowheight}{0.5ex}
\begin{bNiceMatrix}[nullify-dots, xdots/shorten=0.5em] CA^{-1}B & CA^{-2}B & \Cdots & CA^{-p}B & CA^{-(p+1)}B \\
0 & CA^{-1}B\strut & \Cdots & CA^{-(p-1)}B & CA^{-p}B \\
\Vdots & \Ddots & \Ddots & \Vdots & \Vdots \\ \noalign{\kern -6pt}
& & & CA^{-1}B & CA^{-2}B \\
0 & \Cdots & & 0 & CA^{-1}B \end{bNiceMatrix}$

\end{document}

The same with \setlength{\extrarowheight}{1.3ex} instead of 0.5ex: