The phrase “Bellman Operator” is part of the definition! It must appear in it, but earlier.
\documentclass{article}
\usepackage{amsmath,amssymb,amsthm}
\theoremstyle{definition}
\newtheorem*{definition}{Definition}
\begin{document}
\begin{definition}
The \emph{Bellman Operator for $V^{\pi}$} is defined as
$T^{\pi}\colon\mathbb{R}^{|S|} \to \mathbb{R}^{|S|}$ by
\[
(T^{\pi}V^{\pi})(s) =
\sum_{a \in A} \pi(a|s) \Bigl[
R(s,a) + \gamma \sum_{s' \in S} P(s' \mid s,a) V^{\pi}(s')
\Bigr]
\]
It maps value functions to value functions.
\end{definition}
The Bellman operator for $V^{\pi}$ defined above has the following properties:
\begin{itemize}
\item $V^{\pi}$ is a \emph{fixed point} for $V^{\pi}$, which means
that $T^{\pi}V^{\pi} = V^{\pi}$;
\item $T^{\pi}$ is linear;
\item if $0<\gamma<1$ then $T^{\pi}$ is a \emph{maximum-norm contraction}.
\end{itemize}
\end{document}
You may not want the line \theoremstyle{definition}
before defining the definition
environment; in this case the text will be italicized, except for the argument of \emph
which would be in upright type. Usually definitions are typeset in upright type and only the defined term is in italics.
Points to note:
Never leave a blank line before a display
Think twice before leaving a blank line after a display, which means starting a new paragraph
Never use $$
The clause “It maps…” should go after the definition
The text about the properties is a new paragraph, so it should be indented
There's absolutely no need to use \newtheorem
in front of every such environment
The enumerate
or itemize
environment should not have a blank line in front of it, because it's introduced by a clause
It's “which means”, not “that means”
Here's the rendering.

If you feel that \mid
leaves too wide a space around the |
, at least use \,|\,
(and better define a macro).
By the way, I can't make sense of such a definition; please, check the part $V^{\pi}$ is a \emph{fixed point} for $V^{\pi}$
, because it doesn't seem to mean anything sensible.
\item[]
. You have text, displayed math, another paragraph and a list of items. You should put a period to end the math. If you don't want another paragraph after it, just remove the blank line after the math. Also, don't use double dollar, use\[ \]
.\documentclass{...}
and ending with\end{document}
instead of code snippets. But apart from that, after the definition a new paragraph is issued. I think this question covers your case. Or simply add\noindent
after the definition environment.\end{BellOpV}
it is not part of definition. Also, note that it is not italicized.