You should not subvert the \contentsname for changing the formatting, it is merely meant to hold the actual name of the table of contents, and as you experience it is used more than one place. What you see on the second page is the running head.
The formatting for the heading of tableofcontents, and other such lists, is controlled by an internal command called \@makeschapterhead. You can redefine as follows to get bold sans-serif for the headings of all of these lists:
\documentclass{amsbook}
\makeatletter
\def\@makeschapterhead#1{\global\topskip 7.5pc\relax
\begingroup
\fontsize{\@xivpt}{18}\bfseries\sffamily\centering
#1\par \endgroup
\skip@34\p@ \advance\skip@-\normalbaselineskip
\vskip\skip@ }
\makeatother
\begin{document}
\tableofcontents
\chapter{Test}
\end{document}
Should you just wish to change the formatting for the table of contents then you can use the following which switches to a custom style just for the \tableofcontents command:
\documentclass{amsbook}
\makeatletter
\def\@maketochead#1{\global\topskip 7.5pc\relax
\begingroup
\fontsize{\@xivpt}{18}\bfseries\sffamily\centering
#1\par \endgroup
\skip@34\p@ \advance\skip@-\normalbaselineskip
\vskip\skip@ }
\def\tableofcontents{{\let\@makeschapterhead\@maketochead\@starttoc{toc}\contentsname}}
\makeatother
\begin{document}
\tableofcontents
\listoffigures
\chapter{Test}
\end{document}

