# Indicating matching brackets/parentheses in TeX output

I often have cause to typeset things like this in TeX:

When reading expressions like the one above, it's very difficult to tell which brackets match each other. I'd like to be able to define variants of '(' and ')' that were typeset differently depending on the nesting level, to make it easier to match up the parentheses.

I can envisage doing this by introducing commands that increment/decrement some counter (to track the nesting level), and then use the current value of the counter to select the appropriate typesetting mechanism. But I have the impression that using a counter in that way is not very TeXish. What is the best way to do this?

It's not central to the question, but the code that generated the excerpt above is:

\documentclass[a4paper]{article}
\usepackage{libertine}
\begin{document}
$\textrm{subset}(\textrm{applyfn}(\textrm{inverse}(\mathbf{f}),\textrm{union2}(\mathbf{A},\mathbf{B})),\textrm{applyfn}(\textrm{union2}(\textrm{applyfn}(\textrm{inverse}(\mathbf{f}),\mathbf{A}),\textrm{inverse}(\mathbf{f})),\mathbf{B}))$
\end{document}


A solution that involved replacing ( and ) in that with commands would be fine.

Incidentally, editors sometimes have a facility called 'rainbow parentheses' which achieves the same effect; cf.

I don't actually want to use colour, because it's too garish (and I don't want to be forced to use a colour printer), but the desired effect is essentially the same. Suggestions on subtler alternatives to colour would be very welcome.

Edit: greyscale (using Ryan Reich's method) is pretty but ineffective:

underset numbers (a little distracting):

underlines (easy matching but distracting):

-
Can you please tell us something about the samples, where they come from, what the syntax uses/allows...? I came up with using different sized parenthesis, indenting (would really blow up your code size wise i asume), subscripting and replacing the parentheses with different glyphs (braces, brackets, \langle) so far. The replacement seems problematic if your language already assigns meaning to the new symbols or the usable ones are too far of to be recognized as paranthesis like objects. –  Max Dec 30 '12 at 22:02
For readability I suggest prettyprinting with proper indenting rather than different grouping characters. Many code editors will do this automatically. Perhaps there are TeX macros as well. See tug.org/TUGboat/tb15-3/tb44doumont.pdf and www.tug.org/TUGboat/tb19-3/tb60wolin.pdf (although what you need is more like a LISP prettyprinter). –  Ethan Bolker Dec 30 '12 at 22:08
@Max: the TeX I gave above was generated by a program that takes bits of mathematical language and translates them into first order logic. I did consider alternating between square and round brackets, but I was worried that it would distract readers more than it helped. –  Mohan Dec 30 '12 at 22:13
well let's consider dem one by one then. I guess the nesting level is too deep for glyph size to make sense, that already is problematic in normal math code. Different glyphs and indenting would be confusing to big. Leaves the sub/superscripting. How about different (50 :P) shades of grey, but i guess you'd have to apply that all of the enclosed symbols of the same indentation level, so the eye gets a chance to pick those subtleties up. –  Max Dec 30 '12 at 22:20
@Max Shades of grey was the first thing I was going to try! –  Mohan Dec 30 '12 at 22:25

Using a counter in this way is actually very TeXish. I (like your commenters) don't know exactly what scheme will work for you, but here is how to implement something like what you want. Just change what \countlparen and \countrparen do in order to suit your own needs. I admit that as it stands it looks rather ugly. This is a very simple solution: it counts nesting levels but doesn't distinguish braces at a given level, so you still have to do some parsing yourself.

\documentclass{standalone}
\newcounter{parens}
\def\countlparen{%
}
\def\countrparen{%
}
\let\lparen(
\let\rparen)
\begingroup
\catcode(\active
\catcode)\active
\gdef\countparens{%
\let(\countlparen
\let)\countrparen
}
\endgroup
\newenvironment{nested parentheses}
{%
\catcode(\active
\catcode)\active
\countparens
\setcounter{parens}{0}%
}
{}
\begin{document}
\begin{nested parentheses}
$f(g(x)h(k(x)))$
\end{nested parentheses}

f(g(x))

\begin{nested parentheses}
f(g(x)h(k(x)))
\end{nested parentheses}
\end{document}


-
You can have spaces in environment names!? –  Seamus Dec 31 '12 at 11:33
@Seamus: I left that in there just to get that reaction! –  Ryan Reich Dec 31 '12 at 15:19