Reading this site for the last nearly two years I've learned a lot about LaTeX3 (which I had previously understood to be a defunct pipe dream) and LuaTeX (of which I was previously totally unaware). As I have said before, TeX itself is a bad programming language, though a good typesetting engine, but even as a typesetting engine, it is hobbled by its limitations as a language. What I mean by both of these, in short, is that it does not provide an easy way to manipulate its own representations of data. Others have also had this problem, and sometimes it can be solved with LuaTeX, which opens up a lot of the internals to the programming environment.
Now, as I understand it, LaTeX3 also provides a lot of programming tools within TeX, with the aim of making package creation and maintenance more robust. Joseph Wright has explained this in at least one place (and therefore probably many more). These are "algorithmic" tools, while the direction of LuaTeX development appears to be "data structure" tools. Of course, Lua is also a general-purpose programming language and as such provides perfectly good algorithmic tools as well.
Why use LaTeX3 programming over Lua, when LuaTex is available?
I understand that LuaTeX is very much experimental now, but then, so is LaTeX3. I also understand that as a result, LuaTeX is not necessarily used by anyone "ordinary", but then, it is intended as the ultimate successor to pdfTeX, which is now the default engine in the major TeX distributions. For all the reasons that TeX is a terrible programming language, LaTeX3 will still suffer from irritating expansion issues, lack of arithmetic capabilities, poor (or at least poor man's) variables, and so on. However, Knuth himself has written that he would have used a better language if one had been available. Now that "we" are inserting Lua into the TeX executable it appears that this alternative has been identified (ironically, that link is about PyTeX, based on Python).
\directlua is feasible, I don't see why it is not better in every way simply to make all "function" macros (those that do not directly store text) hook into that one primitive and then do all their programming in Lua. Loops, variables, recursion, and so on can all be done in a comprehensible way. A lot of effort in the development of LaTeX3 appears to be in managing the programming language and its use. So, I ask again:
Why use LaTeX3 to program at all? Why not turn its development into the development of a LaTeX-style document design library for LuaTeX, written in Lua?
This suggests a related question to me:
Is the separation of LuaTeX and LaTeX3 a result (or at least an artifact) of the non-communication among developers that Ahmed Musa described in his comment to this answer? What kind of cooperation is there between these two projects to reduce duplication of effort?
(An even more fantastical notion is to implement every primitive, except
\directlua itself, in terms of Lua and various internal typesetting parameters, thus completely divorcing the programming side of TeX from the typesetting side. Then TeX itself would be a "macro package" on top of LuaTeX and one would not be bound to its strange design choices. An answer to why that should or should not be done might be too long for me to request, however.)
I don't personally think that this is too bloggy for this site, and I hope others agree. At least, given the confluence of expertise here, it seems to be the best place to ask the question.
I have accepted Frank Mittelbach's answer, though all the answers are extremely informative, because I think it hits all the main points most directly. One thing that neither it nor Joseph Wright's answer quite manages to come out and say directly is that ConTeXt does do the Lua-programming thing at least to some extent, and it does not solve all algorithmic problems because LuaTeX does not actually rewrite the underpinnings of TeX so much as stitch in a Lua layer that can interact with them.
I guess the one-line answer to my question is that my expectations of LuaTeX are much more grand than its reality, much like my "fantastical" idea.