This is quite easy with xparse
and \NewDocumentCommand
, having two optional arguments, however, you need to check for their existence.
Please note that oo
could be replaced by gg
to allow for optional {}{}
arguments, but that's not recommended, but see below in this answer as an alternative.
\documentclass{article}
\usepackage{xparse}
\NewDocumentCommand{\com}{oo}{%
\lambda%
%Check if the first arg is given
\IfValueT{#1}{%
% Now check if has a 2nd argument as well.
\IfValueTF{#2}{% Yes, the 2nd one is present, use {f}(x) style
\{#1\}(#2)%
}{% No, use [f] style
[#1]%
}%
}%
}
\begin{document}
$\com$
$\com[f]$
$\com[f][x]$
\end{document}

Update with gg
type (use it with care!) and another optional argument to use that instead of \lambda
\documentclass{article}
\usepackage{xparse}
\NewDocumentCommand{\com}{oo}{%
\lambda%
%Check if the first arg is given
\IfValueT{#1}{%
% Now check if has a 2nd argument as well.
\IfValueTF{#2}{% Yes, the 2nd one is present, use {f}(x) style
\{#1\}(#2)%
}{% No, use [f] style
[#1]%
}%
}%
}
\NewDocumentCommand{\comother}{O{\lambda}gg}{%
#1%
%Check if the first arg is given
\IfValueT{#2}{%
% Now check if has a 2nd argument as well.
\IfValueTF{#3}{% Yes, the 2nd one is present, use {f}(x) style
\{#2\}(#3)%
}{% No, use [f] style
[#2]%
}%
}%
}
\begin{document}
$\com$
$\com[f]$
$\com[f][x]$
$\comother$
$\comother{f}$
$\comother{f}{x}$
% Now with \beta instead of \lambda
$\comother[\beta]$
$\comother[\beta]{h}$
$\comother[\beta]{h}{y}$
\end{document}

$\lambda \{f\}(x)$
?xparse
.\com
=\lambda
;\com[f]
=\lambda[f]
;\com{f}(x)
=\lambda\{f\}(x)
. That way your code resembles the output and is easier to understand.