You want a properly defined array using \scriptstyle
. The explain
environment takes as argument the number of columns after the colon.
\documentclass[a4paper,12pt]{book}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{tikz}
\usepackage{array}
\usepackage{booktabs}
\newcolumntype{R}{>{\scriptstyle}r}
% code from: https://tex.stackexchange.com/a/304891/101651
\newcommand*\circled[1]{%
\tikz[baseline=(A.base)]{\node[circle,draw,inner sep=1](A){$#1$};}%
}
\newenvironment{explain}[1]
{%
\renewcommand{\arraystretch}{0.7}%
\setlength{\aboverulesep}{0.5\aboverulesep}%
\setlength{\belowrulesep}{0.5\belowrulesep}%
\begin{array}{@{} R @{\scriptstyle\;:\;} *{#1}{R} @{}}
}
{\end{array}}
\begin{document}
\[
\begin{pmatrix}
2 & 1 \\
1 & -5\\
3 & \circled{7}
\end{pmatrix}
\xrightarrow[
\begin{explain}{2}
-12R_{1} & -24 & -12\\
R_{3} & 3 & 7\\
\cmidrule(l){2-3}
\text{(new) } R_{3} & -21 & -5
\end{explain}
]{-12R_1+ R_3 \to R_3}
\begin{pmatrix}
2 & 1 \\
1 & -5\\
-21 & \circled{-5}
\end{pmatrix}
\]
\end{document}

An alternative definition for explain
is
\usepackage{environ}
\NewEnviron{explain}[1]
{%
\mbox{%
\scriptsize
\setlength{\aboverulesep}{0.5\aboverulesep}%
\setlength{\belowrulesep}{0.5\belowrulesep}%
$\begin{array}{@{} r @{\;:\;} *{#1}{r} @{}}
\BODY
\end{array}$%
}%
}
Without environ
it could be
\newenvironment{explain}[1]
{%
\hbox\bgroup
\scriptsize
\setlength{\aboverulesep}{0.5\aboverulesep}%
\setlength{\belowrulesep}{0.5\belowrulesep}%
$\begin{array}{@{} r @{\;:\;} *{#1}{r} @{}}%
}
{\end{array}$\egroup}
\circled
command?