Their effect is similar but they are differently programmed.
equation* builds on amsmath macros and is therefore more compatible to other amsmath environments and commands.
For instance, within LaTeX's
displaymath environment you cannot use a
split environment, even if you loaded amsmath. But within
equation* it's possible.
\displaymath such that it uses
equation*. That means, after you loaded amsmath, there won't be a difference any more between those two environments. Further, LaTeX's
displaymath is fragile, amsmath makes it robust.
To see that, here are two lines of
And these two lines are at the very end of
To sum up, if you use amsmath, which is very recommendable for math texts, you don't need worry about