your approach to show two sub figures is at least unusual, frankly said wrong:
- you load obsolete package
subfigure
but than not use it. loading it is not sufficient, you should use its environment in figure composition and than compile document at least twice
- package
subfigure
is deprecated. use subfig
instead it or subcaption
(which is more powerful).
- references work on labels, which follows captions. that is not a case in your code example
- it is not clear, what is purpose of tables used in your images
see if the following example works for you (i didn't use thesis
document class, since i don't know which one you use. but this not influence on obtained result). after two compilation you will obtain working references
\documentclass{report} % <--- instead of "thesis" which i haven't
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption} % <--- it is not compatible with "subfigure"
% with it you had to replace "subfigure"
\usepackage{lipsum} % <--- for dummy text
\begin{document}
\lipsum[1]
\begin{figure}[htb] % <---
\centering
\begin{subfigure}[b]{0.6\linewidth} % <---
\includegraphics[width=\linewidth]{example-image-duck}
\caption{Enrollment phase} % <---
\label{fig1.1a:chap1}
\end{subfigure} % <---
\begin{subfigure}[b]{0.6\linewidth} % <---
\includegraphics[width=\linewidth]{example-image-duck}
\caption{Verification phase} % <---
\label{fig1.1b:chap1}
\end{subfigure} % <---
\caption{Mode of operations in a biometric authentication system}
\label{fig1.1:chap1}%
\end{figure}
Test of references: see subfigures \ref{fig1.1a:chap1} and \ref{fig1.1b:chap1} in figure \ref{fig1.1:chap1}.
\end{document}

with % <---
are indicated changes in comparison to your document example. if you like to have wide images, than change width of subfigure
to desired width.
addendum:
you may be lucky and will can use subfigure
package ... :-) :
\documentclass{report}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subfigure} % <---
\usepackage{lipsum}
\begin{document}
\lipsum[1]
\begin{figure}[htb] % <---
\centering
\subfigure[Enrollment phase \label{fig1.1a:chap1}]%
{
\includegraphics[width=0.6\linewidth]{example-image-duck}
}
\subfigure[Verification phase \label{fig1.1b:chap1}]%
{
\includegraphics[width=0.6\linewidth]{example-image-duck}
}
\caption{Mode of operations in a biometric authentication system}
\label{fig1.1:chap1}%
\end{figure}
Test of references: see subfigures \ref{fig1.1a:chap1} and \ref{fig1.1b:chap1} in figure \ref{fig1.1:chap1}.
\end{document}
result is similar as before, however you may experienced new problems with it.

\documentclass
and ending with\end{document}
, with the needed packages.subfigure is obsolete. use
subfig` insteadit orsubcaption
(which is more powerful). it load is not suficient, you should use its macro in table composition and than compile document at least twice.