I truly believe nothing comes close to TeX (and Friends, e.g. LaTeX, ConTeXt) when it comes to output quality.
Other DTP programs such as Indesign (Adobe), QuarkXPress (Quark) or Scribus (open source) offer a graphical interface and hence an easier learning curve, but they don't match up in terms of quality. Amongst many, let me just point the two most obvious quality advantages of TeX:
Typesetting maths correctly is very complex (symbols need to adjust in size, spaces are very specific, etc.), and only TeX gets it right. There are probably even better examples, but try obtaining something like the following in InDesign:
Line and page breaking
Tex's algorithm is very complex and takes into account possible hyphens, widows, orphans, etc. to produce an output that is more elegant.
The best showcase of this I have found is a comparison posted by Roel Zinkstok of Zink Typography, reproduced below. On the comparison, Roel indicated with red dots lines with inter-word spacing that is out of proportion, a cardinal sin for typography, while the red circles indicate hyphens, which should be minimized: