The answer to your question strongly depends on whether the math-mode numerals appear to be (more or less) the same as text-mode numerals, for the font you use. If the two sets of numerals happen to be exactly the same, nobody will ever be able to tell unless they have access to the source code...
However, if they differ noticeably in appearance -- say, because "oldstyle" numerals are used in text mode, which in general should not be used in mathematical expressions -- you obviously need to be careful to distinguish between the two types of numerals.
The following is an excerpt from Knuth's article "Typesetting Concrete Mathematics" (TUGBoat, 1989) [Aside: The textbook Concrete Mathematics was typeset using
Concrete Roman as the text font and
Euler as the math font]:
There was, however, one significant difference between typing the manuscripts for Concrete Mathematics and for The Art of Computer Programming, caused by the fact that the Euler numerals 0123456789 are distinctly different from the numerals 0123456789 in ordinary text. [...]
This experience ... taught me that there is a useful and meaningful distinction between text numerals and mathematical numerals. Text numerals are used in contexts like '1776' and 'Chapter 5' and '41 ways', where the numbers are essentially part of the English language; mathematical numerals, by contrast, are
used in contexts like 'the greatest common divisor of 12 and 18 is 6', where the numbers are part of the mathematics. [...]
Equation numbers presented us with one of the most perplexing design questions. Should those numbers be typeset in Euler or cast in Concrete? After several experiments we hit on a solution that must be right, because it seems so obvious in retrospect: We decided to set equation numbers in an "oldstyle" variant of Concrete Roman ...
What's the upshot of all this? I'd say it's the following: even if at present you're using text and math fonts that have identical-looking numerals, you should still be willing to make the extra effort today to set those numerals that are parts of math expressions in math mode -- as long as there is a chance that you might wish to re-publish your piece one day using fonts for which the math- and text-mode numerals are no longer the same.
What, then, are instances of when you should use text-mode numerals? In addition to the examples given in the preceding quote, I'd also nominate the following: page numbers; numbers of chapters, sections etc; dates (December 25, August 1, ...), and (depending on the circumstances) equation numbers. Obviously, this short list is not meant to be exhaustive.
Addendum: I just noticed that the first two up-votes to this answer earned me my 10,000th point in the TeX.SE group. Thanks everyone!! :-)