I suggest using the amsmath command \genfrac which will resize as follows:

\documentclass{article}
\usepackage{amsmath}
\newcommand{\test}[3]{\left( \genfrac{}{}{0pt}{}{#1}{#2}\,\middle\vert\, #3\right)}
\begin{document}
\begin{equation*}
\test a b c
\quad \scriptstyle \test{a}{b}{c}
\quad \scriptscriptstyle \test{a}{b}{c}
\end{equation*}
\( \test a b c \)
\end{document}
\genfrac takes six arguments, the first two are delimiters around the "fraction", then next is the thickness of the horizontal dividing line, set to 0pt here to make it invisible, and the last two are the numerator and denominator of the fraction.
If you want to have good control over the enclosing brackets (...) then an alternative definition can be given using the mathtools package:

\documentclass{article}
\usepackage{mathtools}
\DeclarePairedDelimiterX{\test}[3]{(}{)}{\genfrac{}{}{0pt}{}{#1}{#2}\,\delimsize\vert\, #3}
\begin{document}
\begin{equation*}
\test* a b c\quad \scriptstyle \test* a b c\quad
\scriptscriptstyle \test* a b c
\end{equation*}
\( \test a b c \)
\begin{equation*}
\test*{a^X}{b_Y}{c_Z} \quad \test[\Bigg]{a^X}{b_Y}{c_Z}
\quad \test[\bigg]{a^X}{b_Y}{c_Z}
\quad \test[\Big]{a^X}{b_Y}{c_Z}
\quad \test[\big]{a^X}{b_Y}{c_Z}
\quad \test{a^X}{b_Y}{c_Z}
\end{equation*}
\end{document}
This time you have use \test* for the version with automatic sizing of brackets, \test just gives you ordinary brackets, but you also get the possiblities \test[\big], \test[\Big] etc. where the brackets sizes are explicitly specified.