This code will make algorithm number be section number . algorithm number

\documentclass[12pt,a4paper]{article}
\usepackage[ruled, vlined, linesnumbered, nofillcomment]{algorithm2e}
\usepackage{algpseudocode}
%****************************************** added <<<<<<<<<<<<<<
\makeatletter
\renewcommand{\fnum@algocf}{\AlCapSty{\AlCapFnt\algorithmcfname\nobreakspace\thesection.\thealgocf}}%
\makeatother
%******************************************
\begin{document}
\setcounter{section}{3}% test section numbers
\setcounter{algocf}{5}% test algo numbers
\section{A section}
\begin{algorithm}
\caption{Name of the algorithm}
test1\\
test2
\end{algorithm}
\end{document}
The caption separator can be controlled using \SetAlgoCaptionSeparator{sep}
(default :
)
If you want to reset the algorithm counter at the beginning of a section add to the preamble
\let\oldsection\section
\renewcommand{\section}{\setcounter{algocf}{0}\oldsection}
OPTION
This is another option using the dcounter
package, using the section number and also will reset the algorithm counter starting a new section of so desired.

\documentclass[12pt,a4paper]{article}
\usepackage[ruled, vlined, linesnumbered, nofillcomment]{algorithm2e}
\usepackage{algpseudocode}
\SetAlgoCaptionSeparator{} % default :
\usepackage{dcounter} % added <<<<<<<<<<<<<<<<<
\countstyle{section}%
\DeclareDynamicCounter{algocf}% reset the algorithm counter starting a new section
\begin{document}
\setcounter{section}{3}% test section numbers
\section{A section}
\begin{algorithm}
\caption{Name of the algorithm 1}
test1\\
test2
\end{algorithm}
\begin{algorithm}
\caption{Name of the algorithm 2}
test1\\
test2
\end{algorithm}
\section{Another section}
\begin{algorithm}
\caption{Name of the algorithm 3}
test1\\
test2
\end{algorithm}
\end{document}