This is something that I first noticed while reading some physics textbooks that are on the slightly older side. Many authors use brackets, and sometimes even braces, to avoid having nesting parentheses. For example:

enter image description here


enter image description here

Is anybody aware of an existing convention for this? I see that many authors don't do this, but I personally find this kind of notation much easier to read.


It is a convention. Where it may be confusing is if you see something like tan(x (1 - x)) it may seem that the convention is not being adhered to but here the outer parentheses indicate that the expression is the argument to the function tan while the inner parenthesis represent a grouping for associative purposes. Looking through the math books I had within reach, I found uses of brackets for second-level grouping in Spivak's Calculus and Gallian's Contemporary Abstract Algebra. I could find no nested parentheses at all in Discovering Number Theory and its worth noting that it took a while to find nested parentheses in Spivak and Gallian. This may be the root of why you hadn't noticed this previously—in most mathematics writing, nested parentheses just aren't that common (with the exception of the tan example I gave at the beginning of my writing. The mathematics of physics, on the other hand, does seem to end up with more of this sort of grouping which could be why you're noticing it more there. I'd note also that both Spivak and Gallian used parentheses rather than braces for the third-level nesting and in one case I spotted brackets in Spivak where parentheses would be appropriate (perhaps a typo resulting from a late edit in the formula).


I would post this as a comment but I don't have enough reputation - sorry.

If it's helpful, there's a good discussion on the Mathematics Stack Exchange, at https://math.stackexchange.com/questions/1875182/how-to-denote-4th-level-of-nested-parentheses/1875207, discussing exactly the conventions of nested brackets.

To summarise, there doesn't appear to be any convention, rather use what is most clear to the reader - whether that's different sized brackets, different bracket types, colours, or something else.


Technically yes there is a convention. https://en.wikipedia.org/wiki/Bracket#Uses_in_mathematics https://www.thoughtco.com/parenthesis-braces-and-brackets-2312410

Not the answer you're looking for? Browse other questions tagged or ask your own question.