It is not to do with maths mode. Here's a simpler example:
This gives an error:
! TeX capacity exceeded, sorry [input stack size=5000].
! ==> Fatal error occurred, no output PDF file produced!
The definition of
\robin is fine. But then it is redefined in terms of...
\robin. So when TeX tries to expand
\robin, it tries to expand
\robin~Hood. So it starts by trying to expand
\robin. Hence it tries to expand
\robin~Hood~Hood and starts by trying to expand
\robin and... ad infinitum. It cannot manage infinite loops, so it declares a fatal error when it runs out of capacity. You have to see its point.
In this kind of case, you need to save the old definition first so you can use the old one in the definition of the new one:
That said, I think it unwise to redefine
\in and I suspect that you are not using the best environment for the maths you are typesetting if you are tempted to redefine it in the way you describe. However, since your actual example doesn't show why you are so tempted, a bit more context would be needed to say anything further about this.