You can render DVI really fast.
My custom, plain TeX, DVI renderer (which is essentially a combination of
dvipng for rendering and
KerTex for compiling
.tex files) runs at ~60 fps for single-page documents (1920x1080), and the DVI renderer alone runs at ~120 fps (1920x1080) and ~250 fps (960x1080) for more pages. And that's that pure CPU rendering, single-threaded.
So not only is there a huge advantage in pure DVI; there's also a huge advantage is plain TeX. (Eventually you'll probably want to extend it, but it should be done without the massive LaTeX bloat. You can get a working TeX distribution for ~6 MB. Here's some of the stuff you can do with such a distribution. I haven't figured out a nice way of inserting images, though.)
A major disadvantage of DVI is that, as far as I know, it doesn't support embedding the fonts in the file itself. (You should be able to embed them yourself, but, if DVI doesn't allow that, then that'd be a custom extension.)