I suggest that the answer to your question significantly varies with what other features of the text coexist with natural language on the page, and with the content of the text. In other words, what is the background to the text, in the sense of these other objects, and what information is conveyed by the text.
Font size is not merely determined by size of the paper.
For example, if you have mathematics, I have found 55 to 57 characters, including space and punctuation, per line. Books with the usual 65 to 67 characters per line are much LESS pleasing to read, especially when equations are periodically littered throughout the text. The font size is appropriate to realize this.
More abstract technical literature (A) often uses 55 to 57, if mathematical in content. Specialized mathematical literature (B) uses 65 to 67. Journals often go 77 to 79.
Fewer characters requires larger characters; try to font size and count the characters, INCLUDING SPACES AND PUNCTUATION, in typical sentences.
Insert a few short equations and you'll see how nicer the larger (but not too large). Vary the count slightly to make the text coherent with the equations, so that nothing JUMPS OUT in contrast and irritates. That's the key to stress-free reading.
I seriously recommend going for 55 to 57, in the memoir class, since you have control over this (as opposed to where you submit to a journal).
If you have no images or equations, you can go up to 65 to 67 (B), as is now usually the case, but I also find non-mathematical text more pleasant reading with 57 to 59, which is closer to (A).
This is partly a matter of taste, so pick up a non mathematical book whose typography you enjoy and count the measure. Do the same for a mathematical book.
A paperback novel, for comparison, has a smaller page and smaller font, but it usually has 63 to 65 characters per line. Again, for comparison, a larger book bound in leather, a theoretical text, from the 1870's has 45 to 47, on a page, and slightly larger margins than conventional; another book, with identical page size, from the 1970's, has 65 to 67, again. This however is the limit, I think, to ease of readability. Ever notice why you slow down while reading journal articles? Neither contains equations. Yet each is substantially larger in size than the paperback novel. A large hardcover quasi-popular scientific monograph, from 2013, contains 53 to 55. It's almost three times the page area of the paperback.
Font size is easy to set in the preamble, and you're no changing margins, I see, so let's analyse the last paragraph.
The basic reason why some typography is hard to read and other typography is not has been considered. Hebb (1949) pointed out, and it is well known today, that the eye does not read the whole page at once, nor does it really scan line by line, but it moves such as to aid what the brain is doing at the moment, which is trying to separate foreground from background, and oscillating back and forth between foreground and background when these are identified, to keep the image clear. The result is integrated and parsed.
The font must not be neither to small, nor too close together, to require careful attention to unconsciously separate the letters. It must also not contrast to much with other components of the text, like mathematical formulae, if any, because if it does, when the formulae are perceived as foreground, the other text becomes background, and vice versa, which is unconsciously distracting and interrupts rapid reading, because the text is no longer parsed together with the equations with which it is, content wise, one thought, one sentence.
In the examples, the content is the deciding factor. If the designer anticipates people to focus on word by word phrasing, like a work of fiction, or a book of philosophy, or an essay chapter, a larger measure (B), would not substantially annoy, regardless of page size, but it does not scale with the page size, only becomes slightly larger.
If the designer anticipates readers to parse the text and think about the concepts in the texts at the same time, so that they are less attentive to the exact wording of the text, a smaller measure is appropriate, because they are not paying so much attention to the words on the page, and the typography must be easier to separate as a whole from its background, since less attention goes to analysing the page, and more to what the text on the page describes.