Please, don't.
Enforce a syntax and stick to it, otherwise users will only get more confused. One way to have multiple arguments without a specific order is to use a key-value syntax:
\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
\tl_new:N \l_recthy_reset_sup_tl
\tl_set_eq:NN \l_recthy_reset_sup_tl \c_novalue_tl
\tl_new:N \l_recthy_reset_sub_tl
\tl_set_eq:NN \l_recthy_reset_sub_tl \c_novalue_tl
\keys_define:nn { rec-thy / REset }
{
, sup .tl_set:N = \l_recthy_reset_sup_tl
, sub .tl_set:N = \l_recthy_reset_sub_tl
}
\prg_generate_conditional_variant:Nnn \tl_if_novalue:n { V } { T , F , TF }
\NewDocumentCommand\REset{om}
{
\group_begin:
\IfValueT {#1} { \keys_set:nn { rec-thy / REset } {#1} }
W \tl_if_novalue:VF \l_recthy_reset_sup_tl { ^{\l_recthy_reset_sup_tl} }
\tl_if_novalue:VTF \l_recthy_reset_sub_tl
{ \c_math_subscript_token {#2} }
{ \c_math_subscript_token {#2, \l_recthy_reset_sub_tl} }
\group_end:
}
\ExplSyntaxOff
\begin{document}
$\REset{e}$\par
$\REset[sub=s]{e}$\par
$\REset[sup=X]{e}$\par
$\REset[sub=s,sup=X]{e}$\par
$\REset[sup=X,sub=s]{e}$\par
\end{document}
But if you find that too wordy for passing the arguments, then you can use xparse's e argument and define a ^ argument and another _ argument, which can appear in any order in the input:
\documentclass{article}
\usepackage{xparse}
\NewDocumentCommand\REset{e{^_}m}
{%
W\IfValueT{#1}{^{#1}}%
\IfValueTF{#2}
{_{#3, #2}}
{_{#3}}%
}
\begin{document}
$\REset{e}$\par
$\REset_{s}{e}$\par
$\REset^{X}{e}$\par
$\REset^{X}_{s}{e}$\par
$\REset_{s}^{X}{e}$\par
\end{document}
And if that didn't please you either, and you insist on keeping the [] and () syntax, then you can make the command have a signature []()[]{} (or ()[](){} or any other combination you like) and then check if the first [] or the second [] was used (and if both were used you can issue an error message):
\documentclass{article}
\usepackage{xparse}
\NewDocumentCommand\REset{od()om}
{%
\IfValueTF{#1}
{%
\IfValueTF{#3}
{\def\recthyOarg{ERROR}%
\PackageError{rec-thy}{Two []-delimited arguments used!}{}}
{\def\recthyOarg{#1}}%
}
{\def\recthyOarg{#3}}%
W\IfValueT{#2}{^{#2}}%
\expandafter\IfValueTF\expandafter{\recthyOarg}
{_{#4, \recthyOarg}}
{_{#4}}%
}
\begin{document}
$\REset{e}$\par
$\REset[s]{e}$\par
$\REset(X){e}$\par
$\REset(X)[s]{e}$\par
$\REset[s](X){e}$\par
\end{document}
All documents produce the same output:
{}are a different catcode than parens()and brackets[]will make this a real challenge. It would be much easier if the syntax were something like\REset{(X)[s]<e>}, with interchangeable units within a single braced argument..