Typically a TeX command have arguments coming after it. But the command
\over, which is used to produce fractions, can access the token before it. How exactly is it implemented and can I define a custom command like that?
It's a TeX primitive so no you can't define commands like it. It's also a pain in the neck and the cause of many of the problems in math mode, as it means that you can not be sure when you first encounter any math mode token what style things will end up in, hence the need for
\mathchoice and various other horrors. If the primitive had had normal prefix syntax like LaTeX's
\frac it wouldn't have been necessary.
It occurs to me that there is one other TeX primitive that can do this:
\lastbox. It doesn't work quite the same; it just grabs the last box produced, if the last thing produced was indeed a box (and if you are not in the "main vertical mode" or math mode). It is somewhat limited, though, since the box is immutable once written, whereas
\over switches its parts to "cramped" style, squeezing superscripts and subscripts. It is important to realize, however, that neither
\lastbox can access previous tokens; once a token is expanded/executed, it is gone. They operate only on lists: math lists or vertical/horizontal lists.
As David Carlisle said, though, its syntax is a poor design decision. For user-visible commands you have the luxury of requiring the author to write the operation before its operands. For internal commands,
\lastbox is useful for picking apart things that were already produced in order to process them using knowledge that's only available after the fact.