Indicating matching brackets/parentheses in TeX output

Question

I often have cause to typeset things like this in TeX:

When reading expressions like the one above, it's very difficult to tell which brackets match each other. I'd like to be able to define variants of '(' and ')' that were typeset differently depending on the nesting level, to make it easier to match up the parentheses.

I can envisage doing this by introducing commands that increment/decrement some counter (to track the nesting level), and then use the current value of the counter to select the appropriate typesetting mechanism. But I have the impression that using a counter in that way is not very TeXish. What is the best way to do this?

It's not central to the question, but the code that generated the excerpt above is:

\documentclass[a4paper]{article}
\usepackage{libertine}
\begin{document}
$\textrm{subset}(\textrm{applyfn}(\textrm{inverse}(\mathbf{f}),\textrm{union2}(\mathbf{A},\mathbf{B})),\textrm{applyfn}(\textrm{union2}(\textrm{applyfn}(\textrm{inverse}(\mathbf{f}),\mathbf{A}),\textrm{inverse}(\mathbf{f})),\mathbf{B}))$
\end{document}

A solution that involved replacing ( and ) in that with commands would be fine.

Incidentally, editors sometimes have a facility called 'rainbow parentheses' which achieves the same effect; cf.

I don't actually want to use colour, because it's too garish (and I don't want to be forced to use a colour printer), but the desired effect is essentially the same. Suggestions on subtler alternatives to colour would be very welcome.

Edit: greyscale (using Ryan Reich's method) is pretty but ineffective:

underset numbers (a little distracting):

underlines (easy matching but distracting):

Answer 1

Using a counter in this way is actually very TeXish. I (like your commenters) don't know exactly what scheme will work for you, but here is how to implement something like what you want. Just change what \countlparen and \countrparen do in order to suit your own needs. I admit that as it stands it looks rather ugly. This is a very simple solution: it counts nesting levels but doesn't distinguish braces at a given level, so you still have to do some parsing yourself.

\documentclass{standalone}
\newcounter{parens}
\def\countlparen{%
    \addtocounter{parens}{1}\lparen\ensuremath{_{\the\value{parens}}}%
}
\def\countrparen{%
    \rparen\ensuremath{_{\the\value{parens}}}\addtocounter{parens}{-1}%
}
\let\lparen(
\let\rparen)
\begingroup
    \catcode`(\active
    \catcode`)\active
    \gdef\countparens{%
        \let(\countlparen
        \let)\countrparen
    }
\endgroup   
\newenvironment{nested parentheses}
{%
    \catcode`(\active
    \catcode`)\active
    \countparens
    \setcounter{parens}{0}%
}
{}
\begin{document}
\begin{nested parentheses}
    $f(g(x)h(k(x)))$
\end{nested parentheses}

f(g(x))

\begin{nested parentheses}
    f(g(x)h(k(x)))
\end{nested parentheses}
\end{document}