How to get nicematrix diagonal dots in the right place

I'm trying to write a matrix with dots in it using the nicematrix package. I'm struggling to get the dots in the right place. My current best attempt looks like this:

The lower triangle of the matrix consists of zeros, which are nicely connected by dots to make a triangle. The diagonal is filled with the term CA^{-1}B. I need those diagonal dots to connect nicely, as they do with the zeros. I think the current diagonal dots that I have marked are aimed towards the rule above the target rule, but I'm not sure of this. An MWE of my best attempt is:

\documentclass{article}
\usepackage{amsmath,nicematrix}

\begin{document}
$\begin{bNiceMatrix}[nullify-dots] CA^{-1}B & CA^{-2}B & \Cdots & CA^{-p}B & CA^{-(p+1)}B \\ 0 & CA^{-1}B & \Cdots & CA^{-(p-1)}B & CA^{-p}B \\ \Vdots & \Ddots & \Ddots & \Vdots & \Vdots \\ & & & CA^{-1}B & CA^{-2}B \\ 0 & \Cdots & & 0 & CA^{-1}B \end{bNiceMatrix}$
\end{document}


How do I make the diagonal dots behave nicely?

What you are seeing is a consequence of the fact that nicematrix draws lines of diagonal dots parallel to each other. The first such line drawn determines the direction (see Technical remarks, Diagonal lines in the manual). Using \extrarowheight, things like \strut in matrix elements and \noalign{\kern -2pt} between lines, one can influence vertical positioning of rows, and thus how lines with diagonal dots connect to matrix elements. Another possibility is to draw lines with diagonal dots in a different order so as to use another direction for them.

Here is my proposal. nicematrix v3.13 or later is required for the xdots/shorten option, but if your nicematrix version is 3.10, 3.11 or 3.12, you can just replace xdots/shorten with dotted-lines-margin.

\documentclass{article}
\usepackage{amsmath}
\usepackage{nicematrix}

\begin{document}

\setlength{\extrarowheight}{0.5ex} \begin{bNiceMatrix}[nullify-dots, xdots/shorten=0.5em] CA^{-1}B & CA^{-2}B & \Cdots & CA^{-p}B & CA^{-(p+1)}B \\ 0 & CA^{-1}B\strut & \Cdots & CA^{-(p-1)}B & CA^{-p}B \\ \Vdots & \Ddots & \Ddots & \Vdots & \Vdots \\ \noalign{\kern -6pt} & & & CA^{-1}B & CA^{-2}B \\ 0 & \Cdots & & 0 & CA^{-1}B \end{bNiceMatrix}

\end{document}


The same with \setlength{\extrarowheight}{1.3ex} instead of 0.5ex:

• In fact, since version 3.13, the option dotted-lines-margin has been renamed xdots/shorten (and xdots/color and xdots/line-style have been created for a full customization of the dotted lines). – F. Pantigny Mar 31 at 11:04
• Many thanks for your input and this great package! I have updated my answer to work with the latest version. Maybe you could have kept the dotted-lines-margin option as a deprecated alias to preserve compatibility. I guess you felt more concerned about avoiding to clutter the code with obsolete, duplicate options... Anyway, you're the maintainer, you decide. :-) – frougon Mar 31 at 11:19
• This is great, thank you both. Your proposal already looks a lot better and you've given me the means to tweak this to my liking. – Iris Apr 1 at 8:54
• You're welcome, glad to be of assistance. – frougon Apr 1 at 9:15