You can't "overload" macros in TeX like functions in other programming languages.
You can either define the macro to use a normal optional argument for one of the two parameters or define a special macro which looks ahead if a opening brace follows. The xparse package can help you defining one:
\documentclass{article}
\usepackage{amsmath}
\usepackage{xparse}
\NewDocumentCommand\Set{mg}{%
\ensuremath{\bigl\{ #1 \IfNoValueTF{#2}{}{\bigm| #2} \bigr\}}%
}
\begin{document}
\[
\Set{A}{B}
\Set{A}
\]
\end{document}
Here the m in the definition stand for mandatory argument and the g for optional argument delimited by a TeX group, i.e. {}.