I'm very new to LaTeX and even though I do understand the power of the system, why is there no single downloadable installer that I can run and install like any other application?
First of all, I think this should be community wiki or closed :-)
TeX is not hard to install at all. Just go to http://tug.org/texlive/ and download the installer. There are other options. We need to know your operating system.
Is TeX itself is hard to install? TeX itself is just a binary, download it and you can use it. If you don't need fonts, don't need additional packages etc. TeX is more then a simple program, it's an ecosystem full of related things. When you install TeX, you get many things that you don't need right now, but probably will need in a month or in a year. Thats the reason why TeX is so large (and rather complex to maintain)
TeX is still evolving. You can get updates very often, and maintaining a good updatable system is pretty hard. Updating your own installation, however, is easy. Both major TeX distributions (TeXLive and MikTeX) provide package managers.
A LaTeX distribution is not a single application but consists from a variety of executables and a large number of packages and other source files written in (La)TeX. As Mike Renfro pointed out a single installer would be a couple of gigabyte in size. In fact a full TeXLive installation is ~2.6 GB in size if you only install the binaries for your platform. If you install really everything (as I recently did), i.e. all binaries for all platforms you have about 4.2 GB.
In fact LaTeX is very easy to install. For Windows simply get the MikTeX installer. MikTeX package manager allows you to easily install any LaTeX package, even automatically once you start using it. The TeXLive installer is text based (at least under Linux, never tried it under windows) and is therefore a little less user-friendly. Besides this it is very easy to use. Also Ubuntu Linux comes with pre-packed (but outdated) TeXLive which can be easily installed over the Ubuntu package manager as usual. See also my answer to What LaTeX system should I use for an Ubuntu/Win7 dual boot environment? for more information.