As you can see from
$ du -sch /usr/local/texlive/2012/texmf-dist/*
the major part is documentation and fonts. If you are satisfied with a single font and without documentation, the distribution can be much smaller. There were several attempts to create such a minimalistic distribution, but I'm not sure if they are still available (edit: I just rediscovered one: grailtex).
And as daleif stated in his comment, you can install a much smaller TeX Live as well.
As real overhead, maybe the
source directory counts, which is also quite large. It consists of sources of most of the stuff in the other directories such as LaTeX packages etc. But this is rather a political than a technical question, therefore I leave it to the TeX Live people to answer it (IIRC they already answered it, maybe on the TeX Live mailing list).