Cascading Style Sheets (CSS) has been evolved to cover many aspects of typesetting and book publishing. Modern browsers already did the heavy lifting part and with the aid of JavaScript publishing can be done in browser.
https://vivliostyle.org/ is a FOSS project that uses CSS + HTML to publish pretty books, and provides both decent math typesetting capabilities via MathML and vertical writing comparable to what pTeX could provide.
There are other CSS publishing tools and they are listed on https://www.print-css.rocks/tools with showcase available.
While their coverage on CSS standards, vendor provided features, prices, and typesetting qualities are all different, some of them are also worth mentioning because in my opinion they also provide equal typesetting quality.
Antennahouse sells the XSL-FO formatter which also supports a good amount of CSS specification, that is used by O’Reilly Media to publish books. The formatter has good multilingual and MathML typesetting support. (The other commercial XSL-FO formatter XEP from RenderX, although it can also produce pretty looking books, does not seem to have math support, so not listed here)
PrinceXML supports a much wider range of CSS standards and has just enough JavaScript support to run MathJax3 (or KaTeX) to get prettier than native MathML result.
Disclaimer on equal typesetting: If taking the mind-blowing Hz and protruding extension implemented by pdfTeX into account, then neither of those mentioned in current answer at current state exceeds or even come to close TeX’s typesetting quality in my honest opinion, but to my knowledge neither do most other typesetting engines except a few the most experimental ones (thus usability is in doubt) mentioned under this question (and automatically excludes InDesign) have implemented them. So as long as it uses dynamic programming to line break text and has the basic math typesetting capability I consider it has equal typesetting.