One needs to declare the object as a closing delimiter.
\documentclass{article}
\usepackage{amsmath}
\NewDocumentCommand{\rbracex}{O{}e{^_}}{%
\mathclose{%
\IfNoValueTF{#2}{% #2 is the possible superscript
\underset{#2}{#1\rbrace}%
}{%
\IfNoValueTF{#3}{% #3 is the possible subscript
\overset{#2}{#1\rbrace}%
}{%
\overunderset{#2}{#3}{#1\rbrace}%
}%
}%
}% matching \mathclose
}
\begin{document}
\[
\lbrace A\rbracex_{i=1}^n
\quad
\big\lbrace A\rbracex[\big]_{i=1}^n
\quad
\Big\lbrace A\rbracex[\Big]_{i=1}^n
\]
\end{document}
I'd avoid this notation with great care.
A possible improvement, prompted by comments by barbara beeton:
\documentclass{article}
\usepackage{amsmath}
\NewDocumentCommand{\rbracex}{O{}e{^_}}{%
\mathclose{%
% let's compute the amount of backing up
\sbox0{\mathsurround=0pt$#1\rbrace$}%
\sbox2{\mathsurround=0pt
$\IfNoValueTF{#2}{% #2 is the possible superscript
\underset{#2}{#1\rbrace}%
}{%
\IfNoValueTF{#3}{% #3 is the possible subscript
\overset{#2}{#1\rbrace}%
}{%
\overunderset{#2}{#3}{#1\rbrace}%
}%
}$%
}% end of \sbox2
\ifdim\wd0>\wd2
% do nothing
\else
\kern\dimexpr(\wd0-\wd2)/2\relax
\fi
\usebox{2}%
}% matching \mathclose
}
\begin{document}
\[
\lbrace A_i\rbracex_{i=1}^n
\quad
\big\lbrace A_i\rbracex[\big]_{i=1}^n
\quad
\Big\lbrace A_i\rbracex[\Big]_{i=1}^n
\]
\end{document}
Well, not really an improvement, I'm afraid.
\strut
.\lbrace A_i\rbrace \rule[-1.5ex]{0pt}{4ex}_{i = 1}^{n}