Package amsmath: Erroneous nesting of equation structures;\n(amsmath) trying to recover with `aligned'

Question

\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\makeatletter
\renewcommand*\env@matrix[1][*\c@MaxMatrixCols c]{%
  \hskip -\arraycolsep
  \let\@ifnextchar\new@ifnextchar
  \array{#1}}
\makeatother
\usepackage{mathtools}
\usepackage{amsfonts}

\usepackage{indentfirst}



\title{Test}
\author{Nguyen Van Manh \\ HE170552}
\date{}

\begin{document}
\maketitle
\section{Ex1}

\textbf{A1}
The augmented matrix of the system:
\[\left[\begin{alignat*}
    1&2&-2&7\\
    -2&1&-1&1\\
    0&3&a&b
\end{alignat*}\right]\]
\[\displaystyle\begin{bmatrix}[cc|c]
    1&2&3\\
    2&-9&17
\end{bmatrix}\]
\end{document}

I get error: Package amsmath: Erroneous nesting of equation structures;\n(amsmath) trying to recover with `aligned'.

What is cause of it? How to fix it?

I was tried:

\[\left[\begin{alignedat}{4}
    &1&&2&&-2&&7\\
    &-2&&1&&-1&&1\\
    &0&&3&&a&&b
\end{alignedat}\right]\tag{1}\]

I don't get the above error at the moment but it looks like it didn't align

Answer 1

In the line

\[\left[\begin{alignat*}

both \[ and \begin{alignat*} initiate standalone display-math mode. That can't go well; hence the error message.

I think what you need to do is replace \begin{alignat*} with \begin{array}{rrrr} and, a few lines down, replace \end{alignat*} with \end{array}.

In view of the fact that you've redefined the low-level \env@matrix macro, you could also replace \left[\begin{alignat*} and \end{alignat*}\right] with \begin{bmatrix}[rrrr] and \end{bmatrix}, respectively.

---

Addendum to address the OP's claim that

\[\left[\begin{alignedat}{4}
    &1&&2&&-2&&7\\
    &-2&&1&&-1&&1\\
    &0&&3&&a&&b
\end{alignedat}\right]\tag{1}\]

should "work". It doesn't -- the alignment of the columns is poor. What would work is

\[\left[\begin{alignedat}{5}
    1&\quad&&2&\quad&&-2&\quad&&7\\
    -2&&&1&&&-1&&&1\\
    0&&&3&&&a&&&b
\end{alignedat}\right]\tag{1}\]

However, I don't think that anyone should prefer this to

\[\begin{bmatrix}[rrrr]
    1&2&-2&7\\
    -2&1&-1&1\\
    0&3&a&b
\end{bmatrix}\]

i.e., the solution I have earlier.