The hyperlinks are set by the macro \NAT@hyper@#1 defined in natbib.sty. Using etoolbox you can hook into the low-level citation command \NAT@citexnum to make \NAT@hyper@ do nothing for non-numeric citations. Here's an example.
\documentclass{article}
\usepackage[numbers]{natbib}
\usepackage[colorlinks]{hyperref}
\usepackage{etoolbox}
\makeatletter
\pretocmd{\NAT@citexnum}{\@ifnum{\NAT@ctype>\z@}{\let\NAT@hyper@\relax}{}}{}{}
\makeatother
\begin{filecontents}{\jobname.bib}
@Book{companion,
author = {Goossens, Michel and Mittelbach, Frank and Samarin, Alexander},
title = {The LaTeX Companion},
edition = {1},
publisher = {Addison-Wesley},
location = {Reading, Mass.},
year = {1994}}
@Book{adams,
title = {The Restaurant at the End of the Universe},
author = {Douglas Adams},
series = {The Hitchhiker's Guide to the Galaxy},
publisher = {Pan Macmillan},
year = {1980}}
\end{filecontents}
\newcommand{\cmd}[1]{\texttt{\textbackslash #1}}
\begin{document}
\noindent
\cmd{citeauthor}: \citeauthor{adams}, \citeauthor{companion} \\
\cmd{cite}: \cite{adams}, \cite{companion} \\
\cmd{citeyear}: \citeyear{adams}, \citeyear{companion} \\
\cmd{citet}: \citet{adams}, \citet[see][p. 20]{adams} \\
multi-\cmd{citet}: \citet{companion,adams} \\
\cmd{citep}: \citep{adams}, \citep[see][p. 20]{companion} \\
multi-\cmd{citep}: \citep{companion,adams} \\
\cmd{citetext}, \cmd{citealp}: \citetext{see \citealp{companion}, or even better \citealp{adams}}
\bibliographystyle{plainnat}
\bibliography{\jobname}
\end{document}

Thanks to Kurt for pointing out that the initial solution didn't work.