If you look at the definition of the classic
style, contained in moderncvstyleclassic.sty
you will find all the macros that control the appearance of the elements of the document. For fonts you'll find:
\renewcommand*{\namefont}{\fontsize{34}{36}\mdseries\upshape}
\renewcommand*{\titlefont}{\LARGE\mdseries\slshape}
\renewcommand*{\addressfont}{\small\mdseries\slshape}
\renewcommand*{\quotefont}{\large\slshape}
\renewcommand*{\sectionfont}{\Large\mdseries\upshape}
\renewcommand*{\subsectionfont}{\large\mdseries\upshape}
\renewcommand*{\hintfont}{}
The one you are trying to alter is \addressfont
so putting
\renewcommand*{\addressfont}{\small\mdseries}
in your preamble should do what you want.