You can have the first two images side by side, but obviously not the last one.
The problem is that you're inserting a space between the subfigures, so the first two cannot fit in a line. Look for %<---
to see where the space was.
For the third picture, it's better to have a paragraph break and some vertical space.
\documentclass{article}
\usepackage{graphicx}
\usepackage{authblk}
\usepackage{float}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{algorithm}
\usepackage{amsmath}
\begin{document}
\begin{figure}
\centering
\begin{subfigure}[b]{0.5\textwidth}
\centering
\includegraphics[width=\textwidth]{histo1.jpg}
\caption{}
\label{fig:y equals x}
\end{subfigure}%<---
\hfill
\begin{subfigure}[b]{0.5\textwidth}
\centering
\includegraphics[width=\textwidth]{histo2.jpg}
\caption{}
\label{fig:three sin x}
\end{subfigure}
\bigskip
\begin{subfigure}[b]{0.5\textwidth}
\centering
\includegraphics[width=\textwidth]{histo3.jpg}
\caption{}
\label{fig:five over x}
\end{subfigure}
\caption{ Histogram Plot of (a) Input Image, (b) Encrypted Image and (c) Decrypted Image }
\label{fig:three graphs}
\end{figure}
\end{document}

If you want the three pictures all on one line, use the textwidth divided by 3.
\documentclass{article}
\usepackage{graphicx}
\usepackage{authblk}
\usepackage{float}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{algorithm}
\usepackage{amsmath}
\begin{document}
\begin{figure}
\centering
\begin{subfigure}[b]{0.33333\textwidth}
\centering
\includegraphics[width=\textwidth]{histo1.jpg}
\caption{}
\label{fig:y equals x}
\end{subfigure}%<---
\hfill
\begin{subfigure}[b]{0.33333\textwidth}
\centering
\includegraphics[width=\textwidth]{histo2.jpg}
\caption{}
\label{fig:three sin x}
\end{subfigure}%<---
\begin{subfigure}[b]{0.33333\textwidth}
\centering
\includegraphics[width=\textwidth]{histo3.jpg}
\caption{}
\label{fig:five over x}
\end{subfigure}
\caption{ Histogram Plot of (a) Input Image, (b) Encrypted Image and (c) Decrypted Image }
\label{fig:three graphs}
\end{figure}
\end{document}

\documentclass{}
and end with\end{document}
, a Minimal Working Example (MWE) please. It makes copy and pasting easier when the code can be compiled and also we do not need to guess what commands or packages you are using.