MusiXTeX is TeX based. Its latest versions also uses the auxiliary program
musixflx that reformats the first attempt made with MusiXTeX in order to adjust the spacing between (groups of) notes or bars in order to completely fill the lines.
It's a very remarkable piece of software, all TeX based apart from that external intervention. Notes are made with characters and very complex scores can be produced. One can find several scores at the Werner Icking Music Archive now merged with the International Music Score Library Project.
LilyPond, on the other hand, has a syntax that is largely inspired by TeX, but that's the only likeness. It's a completely different system and uses its own engine and fonts for typesetting music. According to the Wikipedia article, LilyPond started off from MusiXTeX and this explains the similarities in syntax, but rapidly separated from its progenitor.
It's actively developed and is capable of very high level results, even better than expensive proprietary software (according to a musician friend of mine).
Personal note: I wrote a package for integrating in LaTeX music typeset with a different system (ABC); I tried to adapt it also to LilyPond, but the similarities in syntax are an obstacle. However integration is possible the other way around with scripts such as
I would like to add little bit to the great answer you got from egreg risking to be corrected from many professional musician who hang here and who are much bigger TeX experts than me.
If you look closely well-printed books you will see that certain combinations of letters are treated as a unit. For example fi. In typography such unit is known under the name of ligature. Ligatures occurs where two or more graphemes are joined as a single glyph. Ligatures usually replace consecutive characters sharing common components and are part of a more general class of glyphs called "contextual forms". Unlike written text music scores are notoriously full of ligatures. Just think of whole note vs triplets. Unfortunately when you get deeper into the typography of the music scores you will see that symmetric spacing between musical ligatures usually mean poor typesetting quality. Unlike TeX which prefers symmetric spacing (that is an oversimplification but check about glue in TeX) Lilypond is designed from the ground up to deal with non-symmetric spacing of musical ligatures. Although it is possible to type music in TeX (MusicTeX and MusiXTeX) LilyPond is just more beautiful. People who developed LilyPond used to work on MusicTeX as developers. I will go even further and say that typing music is one of those rare occasions where TeX should not be used.
Disclaimer: I have been using TeX almost all my adult and professional mathematician life (circa 1991) and I stumbled upon Lilypond five years ago when I wanted to typeset few songs for my infant daughters. I am by no means proficient in LilyPond. From the top of my head I think that Lilypond is capable of converting music from a keyboard to scores via midi interface. It is just all together different beast when it comes to music.
Two really good answers. I'd like to add (at the risk of this thread becoming too "bandwagon-y"):
LilyPond really strives to be an independent and dedicated engine for music. The focus is on the manipulation of musical content through macro-processes. This coincides with the out of the box ability to produce self-contained files of PDFs, or other various and sundry files for print without much upstart in the learning curve.
But beyond this, there is an aesthetic advantage that LilyPond wants to strive for that I have not seen in MusiXTeX - To cover the entire spectrum of musical notation with beauty and grace. This shows up in ancient scores, recent developments in OTF fonts for use in LilyPond, and the superior (and less infuriating) process of adapting the graphical output of LilyPond to fit a users needs.