The main paper about the algorithms that TeX uses for laying out text is:
Breaking paragraphs into lines, by Michael F. Plass and Donald E. Knuth, published in Software—Practice and Experience Volume 11 (1981), pages 1119–1184.
You can find the paper online or read its latest version reprinted in the book Digital Typography; you can also search on this site to find answers mentioning it (like this one).
The algorithm is of course described in more detail in the source code — available as a printed book (TeX: The Program), and also available (without a few things like the mini-indices on each page) digitally. Specifically, this starts in Part 38 of the program (section 813), and continues into Part 39. This is also described, to some extent, in The Texbook, though mostly from a “user's” point of view.
Both The TeXbook and the TeX program have the flaw (IMO) that understanding any part of it properly requires you to understand much of the entire thing, so the paper is probably the best place to start.
Of course there is more to text layout than breaking paragraphs into lines (though in the case of TeX you may be surprised to discover how much just boils down to this basic box-glue-penalties model for line breaking, even things that don't seem to be about line-breaking). Everything that TeX does is in the TeX program, though you may want to read the corresponding research papers first before reading the relevant parts of the program.
LaTeX is a set of macros on top of TeX that adds quite a few things, but not much of that (AFAIK) is directly related to the question you seem to be asking, on how text is laid out.