Yes, you can do that, as explained on page 16 of the fancyvrb
documentation:
\documentclass{article}
\usepackage{fancyvrb}
\begin{document}
\begin{Verbatim}[commandchars=\\\{\},
codes={\catcode`$=3\catcode`^=7\catcode`_=8},frame=single]
some code test
$\sum a_i$
\end{Verbatim}
\end{document}

I'd suggest you to switch to listings
and use its mathescape=true
feature:
\documentclass{article}
\usepackage{listings}
\lstset{
basicstyle=\ttfamily,
columns=fullflexible,
mathescape=true,
frame=single
}
\begin{document}
\begin{lstlisting}
some code test
$\sum a_i$
\end{lstlisting}
\end{document}

If you want fancier boxes, then you could use tcolorbox
and its interaction with listings
.