Is there a standalone version of LaTeX that might run of, say, a USB stick? So a version that doesn't need installing. I am looking for something to run under Windows.
Update 2014-02: It has been two years since this response was written. As of February 2014, available packages for USB installation are: TexLive option to run from USB stick (for multiple OSes), MikTex Portable, and TexPortable (both Windows).
I guess you're looking for USBTeX: http://www.exomatik.net/U-Latex/USBTeXEnglish
Update 2014-02: USBTeX seems to be no longer maintained; the above URL does not work anymore.
MikTex Portable is exactly what you want. It is a "portable" version of MiKTeX, one of the major TeX distributions, by the MiKTeX project itself. It's the first thing to try before considering repackaged distributions.
USBTeX is no longer maintained and the link (http://www.exomatik.net/U-Latex/USBTeXEnglish) is broken, so if you still are looking for a portable LaTeX environment for Windows, you could try TexPortable, which you can obtain from here.
Basically it's a combination of MiKTeX, Texmaker and SumatraPDF.
If you're going to give it a try, please let me know if you have any problems.
The Knoppix DVD has
TeXmaker and even
Lyx basically pre-installed. It will run in VMWare, VirtualBox, or any other virtualization software. You can download the ISO image. Be advised this is an entire operating system, which runs from the DVD. You can boot into it outright, or just run it in a Virtual Machine. Order CD/DVD online and other KNOPPIX vendors in addition to vendors at distrowatch can send you a DVD for cheap