The biggest limitations I can think of (strictly in comparison with programs like InDesign or Word) are as follows.
Mind you, I have no idea what consequences any changes would have wrt. computational complexity or output quality of the system as a whole. TeX does the things it does extremely fast and in near optimum quality, so it may well be in the "pareto set", so to say.
But anyway...
Paragraph formatting and column/page breaking are strictly detached from each other.
Consequently, TeX's paragraph optimization algorithm doesn't "know" where a certain line will be positioned on the page and can't take this into account.
This has some dire consequences. Off the top of my head:
- It's awfully hard to make text "flow around" things on the page, especially when page breaks or flush bottom typesetting are involved.
- I can avoid breaking the page at a line containing a hyphen or a widow/orphan, but I can't make a penalty for paragraph breaking to avoid it appearing on this line.
TeX is missing a lot of "meta information" on the page
Things like color, z coordinate, angle, writing direction etc. are somehow 'fiddled' into the boxes making up the page or are just implicit in the order content is output. Most of the info can't be inspected later, and communication between content items ("does this box collide with the other one when it's turned around?") becomes next to impossible.
TeX has a "waterfall" model of page building
Whenever content has left one part of TeXs digestive system for another one, there mostly is no way back. The best we can achieve is to undo everything until a certain stage (for instance, by throwing a box away instead of outputting it) and retrying with different parameters.
If TeX had an object oriented page model where every information could be freely inspected, modified or restructured at any stage and where "typesetting" and "page building" mainly meant to re-structure objects and enrich them with meta information ("where has this paragraph been broken, how much has glue been stretched") which can later be inspected or modified, things would be much much easier.