I would suggest you change the syntax to \sumsquare{a}{}:
This can be coded either this way:
\newcommand{\sumsquare}[2]{%
\ifx\\#2\\%
\ensuremath{#1^2}}%
\else
\ensuremath{(#1+#2)^2}}%
\fi
}
or:
\newcommand{\sumsquare}[2]{%
\begingroup
\def\tempvar{#2}%
\ifx\tempvar\empty
\endgroup
\ensuremath{#1^2}}%
\else
\endgroup
\ensuremath{(#1+#2)^2}}%
\fi
}
If your really want 0 as the no-operant indicator you can define it like this:
\newcommand{\sumsquare}[2]{%
\begingroup
\def\tempvara{#2}%
\def\tempvarb{0}%
\ifx\tempvara\tempvarb
\endgroup
\ensuremath{#1^2}}%
\else
\endgroup
\ensuremath{(#1+#2)^2}}%
\fi
}
(I named the temporary variables this way to avoid \makeatletter. Normally \@tempa and \@tempb are used.)
Explanation:
The \ifx command compares the next to tokens (e.g. macros, characters, ...) if the hold the same definition. In the last example the 0 and the #2 are both defined to a macro each, which are then compared. This requires assignments and is therefore not expandable, i.e. doesn't work inside an \edef.
In the first code \ifx\\#2\\ is used to test if #2 is empty. If #2 contains something, \ifx compares the first token in it with \\, which does not match as long #2 doesn't start with \\. All other tokens are then simply taken as part of the true part and simply discarded with it.
If \\ is a valid value for #2 simply use some other macro like \relax or \@nnil instead.
However, if #2 is empty the expression is reduced to \ifx\\\\, i.e. \ifx compares two \\, which are of course defined identical.