Punctuation that is not part of a formula should go outside the formula:
When \(a>0\), we say that \(a\) is \emph{positive}.
In the interval \([0,1]\) there are infinitely many rational numbers.
It's quite rare that quotes are used as math symbols, but I've seen them in some textbook on mathematical logic. Since '
is used for denoting a prime (dash) in math, one can resort to a different trick: the lines
\DeclareMathSymbol{\mlq}{\mathord}{operators}{``}
\DeclareMathSymbol{\mrq}{\mathord}{operators}{`'}
will define two commands that will give left and right quotes in math and your input can become
Why can't I get $\mlq a\mrq$ in math mode.
but in this case `$a$'
should be probably better.
I've added a companion to \mrq
just for symmetry and, perhaps, greater clarity.