Would it be correct to say that somewhere within Tex, there is a layout algorithm? If there is, is a description of the algorithm and abstractions it consumes and produces available?
A question and answer here address How does Tex actually work, but don't cover the layout process I am hypothesising.
Another question here asks if Tex layout can be decoupled from it's macros, and a comment on that links to Knuth's paper Breaking paragraphs in to lines, which seems like a relevant place to start reading.
Sorry if I've chosen a poor tag. I need to add at least one, and none I've found matching "algorithm" or "layout" seem appropriate.