This is an improvement over John Kormylo's answer -- it works, but has a small quirk: it hard codes the math style.
What does this mean? Consider for example, with that definition:
\[ \mydelim{\frac{x+a}{x+b}} \]
$ \mydelim{\frac{x+a}{x+b}} + \frac{x+a}{x+b} $
the second formula is too large, compared to the one outside the custom delimiter.
One way to fix it is to manually add the \textstyle
command:
$ \mydelim{\textstyle \frac{x+a}{x+b}} \frac{x+a}{x+b} $
(in this case you don't need the \displaystyle
at all)
Another way is to modify the command definition as follows:
\documentclass{article}
\usepackage{tikz}
\usepackage{amsmath} % ← note the added package
\newcommand{\leftdelim}[2]% #1 = height, #2 = depth
{\tikz[baseline]{\draw[color=red,arrows={-latex}](0,-#2)--(0,#1);}}
\newcommand{\rightdelim}[2]% #1 = height, #2 = depth
{\tikz[baseline]{\draw[color=red,arrows={latex-}](0,-#2)--(0,#1);}}
\newlength{\MyHeight}
\newlength{\MyDepth}
\newsavebox{\MyBox}
\newcommand{\mydelim}[1]% #1 = text to be enclosed
{\text{\savebox{\MyBox}{$#1$}% get size of box
\settoheight{\MyHeight}{\usebox{\MyBox}}%
\settodepth{\MyDepth}{\usebox{\MyBox}}%
\leftdelim{\MyHeight}{\MyDepth}%
\,\usebox{\MyBox}\,%
\rightdelim{\MyHeight}{\MyDepth}}}
\begin{document}
\[ \mydelim{\frac{x+a}{x+b}} \]
$ \mydelim{\frac{x+a}{x+b}} + \frac{x+a}{x+b} $
\end{document}
which gives the desired result.
An explanation of the \text
command may be found at rotating - Rotated math with correct font size - TeX - LaTeX Stack Exchange.
\left
and\right
but I'm not sure if it is a good idea... this and this answer might help you.\left
or\right
only if scalable versions of the symbol are available as part of the font.