This is a very important function in the LaTeX kernel. The macro
\@ifnextchar
takes three argument. The first one should be a single token, usually [ but not necessarily.
When the input stream has the following tokens
\@ifnextchar<token>{<true>}{<false>}
TeX will look at the next token (skipping spaces) and compare it to the <token> which is the first argument to \@ifnextchar and will not remove it from the input stream.
If the two tokens coincide (have the same meaning, to be precise), TeX will use the <true> code, otherwise the <false> code.
The old fashioned way to define commands with an optional and a mandatory argument was something like
\newcommand{\xyz}{\@ifnextchar[{\@xyz}{\@xyz[default]}}
\def\@xyz[#1]#2{do something with #1 and #2}
where default is the default value for the optional argument. Nowadays we'd say
\newcommand\xyz[2][default]{something with #1 and #2}
(which eventually will do the same working as the old fashioned definition, but in a safer way). If the call is
\xyz[a]{b}
the test of \@ifnextchar would be true, so TeX would expand this into
\@xyz[a]{b}
because the [ is not removed as would be an argument. With
\xyz{b}
the test would be false, so the tokens would be replaced by
\@xyz[default]{b}
keeping TeX happy with respect to the definition of \@xyz.
The same applies in your case, with the difference that the <false> text has something more in it:
\@ifnextchar[\@myitem{\@noitemargtrue\@myitem[\@itemlabel]}}[aaa] bbb will become
\@myitem[aaa] bbb
\@ifnextchar[\@myitem{\@noitemargtrue\@myitem[\@itemlabel]}} bbb will become
\@noitemargtrue\@myitem[\@itemlabel] bbb
so supplying a suitable argument between square brackets to \@myitem and setting a conditional.
Since the second argument to \@ifnextchar, in this case, is a single token, it doesn't need braces around it. The code
\@ifnextchar[{\@myitem}{\@noitemargtrue\@myitem[\@itemlabel]}}
would be completely equivalent.
See this answer for an explanation of how \@ifnextchar works internally in terms of \futurelet and this other one for a description of \futurelet (both by Martin Scharrer).