If you look at the break down of the big directories on my Linux 2016 install of TeX Live (which is also 4.6 GB)
If you are willing to limited yourself to a few fonts, most of
./fonts can go. If you are happy with online documentation, you can get rid of
./doc. I am pretty sure
./backups are not needed.
You are going to get reduced functionality if you delete things in
./tex (this is where the packages and classes live) and
./bin (this is where the programs live), but there are packages and programs that are rarely used so some space can be saved.