How are commands with an optional argument defined in the LaTeX kernel way?
The key function is \@ifnextchar, which tests the following token (gobbling spaces) and can perform different actions. See Understanding \@ifnextchar for more information.
In your case we want a command \foo that does different things when called as \foo{m1}{m2}, \foo{m1}[o1]{m2} or \foo{m1}{m2}[o2]. The two optional argument are mutually exclusive.
Let's start: the macro \foo will collect the first mandatory argument and test for [:
\def\foo#1{\@ifnextchar[{\foo@firstopt{#1}}{\foo@nofirstopt{#1}}}
It's very important that \@ifnextchar<token>{<true>}{<false>} is at the end.
We now have to define \foo@firstopt, which has to absorb another mandatory argument; the [ is not removed from the main input list, so we can do
\def\foo@firstopt#1[#2]#3{%
<the code for the "o1" case>%
\@ifnextchar[{\foo@badsecondopt}{}%
}
At the end we add code for testing the presence of the second optional argument, to raise an error and remove the offending part:
\def\foo@badsecondopt[#1]{<raise an error>}
Now let's tackle \foo@nofirstopt; we need to check for a trailing optional argument:
\def\foo@nofirstopt#1#2{%
\@ifnextchar[{\foo@secondopt{#1}{#2}}{\foo@nosecondopt{#1}{#2}}%
}
Now it's easy:
\def\foo@secondopt#1#2[#3]{%
<the code for the "o2" case>%
}
\def\foo@nosecondopt#1#2{%
<the code for the "no optional arguments" case>%
}
The description of \newtheorem is a bit more complicated, because the kernel tries to avoid code duplication.
Note how already absorbed arguments can be carried on to the next stage.
How can we do the same with xparse?
\NewDocumentCommand{\foo}{m o m o}{%
\IfNoValueTF{#1}%
{% no o1
\IfNoValueTF{#2}%
{% no o2
<code for the "no optional arguments" case>%
}%
{% o2
<code for the "o2" case>%
}%
}%
{% o1
<code for the "o1" case>%
\IfNoValueF{#2}{<error message>}%
}%
}
The two mandatory arguments are referred to as #1 and #3, the two optional arguments as #2 and #4.
