You might use the G
specifier, which denotes an optional argument in braces:
\documentclass{article}
\usepackage{amsthm,xparse}
\usepackage{lipsum}
\makeatletter
\newtheoremstyle{jessica}% name
{\topsep}% Space above
{\topsep}% Space below
{\normalfont}% Body font
{}% Indent amount (empty = no indent, \parindent = para indent)
{\bfseries}% Thm head font
{}% Punctuation after thm head
{0pt}% Space after thm head: " " = normal interword space;
{%
\makebox[0pt][r]{\makebox[\jessicaindent][l]{#2}}%
#1%
\@ifnotempty{#3}{\textnormal{ (#3)}}%
\hspace{\jessicalabelsep}%
}
\makeatother
\newlength{\jessicaindent}
\newlength{\jessicalabelsep}
\setlength{\jessicaindent}{4em}
\setlength{\jessicalabelsep}{0.5em}
\theoremstyle{jessica}
\newtheorem{jessicacustom}{\jessicacustomname}[section]
\newtheorem*{jessicacustomnonumber}{\jessicacustomname}
\NewDocumentEnvironment{custom}{G{Custom}}
{\def\jessicacustomname{#1}\jessicacustom}
{\endjessicacustom}
\NewDocumentEnvironment{custom*}{G{Custom}}
{\def\jessicacustomname{#1}\jessicacustomnonumber}
{\endjessicacustomnonumber}
\begin{document}
\section{One}
\lipsum[4]
\begin{custom}{Custom Title}[optional words]\label{A}
\lipsum*[2]
\end{custom}
\begin{custom}\label{B}
Even though I've been teaching for n! years, every class is a fresh
adventure -- An adventure. That word is a euphemism for the stark reality,
which is that every class contains some totally shocking development
that I have never seen before and haven't any idea how to cope with.
Just last year, for the first time in many years, I taught a section
of the junior-level mathematical analysis course. You know -- the one
where students meet proofs in analysis for the first time. The one where
students and epsilons meet, eyeball to eyeball, and it isn't the epsilons
that blink. The one where students decide that they really wanted to be
doctors and lawyers after all.
\end{custom}
\begin{custom*}{Notnumbered}[xyz]
This is not numbered.
\end{custom*}
Here are the references: \ref{A} and \ref{B}.
\end{document}

I'd prefer a key-value interface:
\documentclass{article}
\usepackage{amsthm,xparse}
\usepackage{lipsum}
\makeatletter
\newtheoremstyle{jessica}% name
{\topsep}% Space above
{\topsep}% Space below
{\normalfont}% Body font
{}% Indent amount (empty = no indent, \parindent = para indent)
{\bfseries}% Thm head font
{}% Punctuation after thm head
{0pt}% Space after thm head: " " = normal interword space;
{%
\makebox[0pt][r]{\makebox[\jessicaindent][l]{#2}}%
#1%
\@ifnotempty{#3}{\textnormal{ (#3)}}%
\hspace{\jessicalabelsep}%
}
\makeatother
\newlength{\jessicaindent}
\newlength{\jessicalabelsep}
\setlength{\jessicaindent}{4em}
\setlength{\jessicalabelsep}{0.5em}
\theoremstyle{jessica}
\newtheorem{jessicacustom}{\jessicacustomname}[section]
\newtheorem*{jessicacustomnonumber}{\jessicacustomname}
\ExplSyntaxOn
\keys_define:nn { jessica/custom }
{
number .bool_set:N = \l_jessica_custom_numbered_bool,
number .default:n = true,
number .initial:n = true,
nonumber .bool_set_inverse:N = \l_jessica_custom_numbered_bool,
nonumber .default:n = true,
name .tl_set:N = \l_jessica_custom_name_tl,
name .initial:n = Custom,
attr .tl_set:N = \l_jessica_custom_attr_tl,
}
\tl_new:N \l__jessica_custom_temp_tl
\NewDocumentEnvironment{custom}{O{}}
{
\keys_set:nn { jessica/custom } { #1 }
\tl_set_eq:NN \jessicacustomname \l_jessica_custom_name_tl
\tl_set:Nx \l__jessica_custom_temp_tl
{
\bool_if:NTF \l_jessica_custom_numbered_bool
{ \exp_not:N \jessicacustom }
{ \exp_not:N \jessicacustomnonumber }
\tl_if_empty:NF \l_jessica_custom_attr_tl
{ [\exp_not:V \l_jessica_custom_attr_tl] }
}
\tl_use:N \l__jessica_custom_temp_tl
}
{
\bool_if:NTF \l_jessica_custom_numbered_bool
{ \endjessicacustom }
{ \endjessicacustomnonumber }
}
\ExplSyntaxOff
\begin{document}
\section{One}
\lipsum[4]
\begin{custom}[name=Custom Title,attr=optional words]\label{A}
\lipsum*[2]
\end{custom}
\begin{custom}\label{B}
Even though I've been teaching for n! years, every class is a fresh
adventure -- An adventure. That word is a euphemism for the stark reality,
which is that every class contains some totally shocking development
that I have never seen before and haven't any idea how to cope with.
Just last year, for the first time in many years, I taught a section
of the junior-level mathematical analysis course. You know -- the one
where students meet proofs in analysis for the first time. The one where
students and epsilons meet, eyeball to eyeball, and it isn't the epsilons
that blink. The one where students decide that they really wanted to be
doctors and lawyers after all.
\end{custom}
\begin{custom}[name=Notnumbered,nonumber,attr=xyz]
This is not numbered.
\end{custom}
Here are the references: \ref{A} and \ref{B}.
\end{document}

\documentclass{article}
\usepackage{amsthm,xparse,etoolbox}
\usepackage{lipsum}
\makeatletter
\newlength{\jessicaindent}
\newlength{\jessicalabelsep}
\setlength{\jessicaindent}{4em}
\setlength{\jessicalabelsep}{0.5em}
\newtheoremstyle{jessica}% name
{\topsep}% Space above
{\topsep}% Space below
{\normalfont}% Body font
{}% Indent amount (empty = no indent, \parindent = para indent)
{\bfseries}% Thm head font
{}% Punctuation after thm head
{0pt}% Space after thm head: " " = normal interword space;
{%
\makebox[0pt][r]{\makebox[\jessicaindent][l]{#2}}%
#1%
\@ifnotempty{#3}{\textnormal{ (#3)}}%
\hspace{\jessicalabelsep}%
}
\theoremstyle{jessica}
\newtheorem{jessicacustom}{\jessicacustomname}[section]
\newtheorem*{jessicacustomnonumber}{\jessicacustomname}
\NewDocumentEnvironment{custom}{G{Custom}}
{%
\patchcmd{\@thm}{\trivlist}{\list{}{\leftmargin=\jessicaindent}}{}{}%
\def\jessicacustomname{#1}\jessicacustom
}
{\endjessicacustom}
\patchcmd{\endjessicacustom}{\endtrivlist}{\endlist}{}{}
\NewDocumentEnvironment{custom*}{G{Custom}}
{%
\patchcmd{\@thm}{\trivlist}{\list{}{\leftmargin=\jessicaindent}}{}{}%
\def\jessicacustomname{#1}\jessicacustomnonumber
}
{\endjessicacustomnonumber}
\patchcmd{\endjessicacustomnonumber}{\endtrivlist}{\endlist}{}{}
\makeatother
\begin{document}
\section{One}
\lipsum[4]
\begin{custom}{Custom Title}[optional words]\label{A}
\lipsum*[2]
\end{custom}
\begin{custom}\label{B}
Even though I've been teaching for n! years, every class is a fresh
adventure -- An adventure. That word is a euphemism for the stark reality,
which is that every class contains some totally shocking development
that I have never seen before and haven't any idea how to cope with.
Just last year, for the first time in many years, I taught a section
of the junior-level mathematical analysis course. You know -- the one
where students meet proofs in analysis for the first time. The one where
students and epsilons meet, eyeball to eyeball, and it isn't the epsilons
that blink. The one where students decide that they really wanted to be
doctors and lawyers after all.
\end{custom}
\begin{custom*}{Notnumbered}[xyz]
This is not numbered.
\end{custom*}
Here are the references: \ref{A} and \ref{B}.
\end{document}

The same with the key-value interface.
\documentclass{article}
\usepackage{amsthm,xparse,etoolbox}
\usepackage{lipsum}
\makeatletter
\newlength{\jessicaindent}
\newlength{\jessicalabelsep}
\setlength{\jessicaindent}{4em}
\setlength{\jessicalabelsep}{0.5em}
\newtheoremstyle{jessica}% name
{\topsep}% Space above
{\topsep}% Space below
{\normalfont}% Body font
{}% Indent amount (empty = no indent, \parindent = para indent)
{\bfseries}% Thm head font
{}% Punctuation after thm head
{0pt}% Space after thm head: " " = normal interword space;
{%
\makebox[0pt][r]{\makebox[\jessicaindent][l]{#2}}%
#1%
\@ifnotempty{#3}{\textnormal{ (#3)}}%
\hspace{\jessicalabelsep}%
}
\theoremstyle{jessica}
\newtheorem{jessicacustom}{\jessicacustomname}[section]
\newtheorem*{jessicacustomnonumber}{\jessicacustomname}
\patchcmd{\endjessicacustom}{\endtrivlist}{\endlist}
\patchcmd{\endjessicacustomnonumber}{\endtrivlist}{\endlist}
\makeatother
\ExplSyntaxOn
\keys_define:nn { jessica/custom }
{
number .bool_set:N = \l_jessica_custom_numbered_bool,
number .default:n = true,
number .initial:n = true,
nonumber .bool_set_inverse:N = \l_jessica_custom_numbered_bool,
nonumber .default:n = true,
name .tl_set:N = \l_jessica_custom_name_tl,
name .initial:n = Custom,
attr .tl_set:N = \l_jessica_custom_attr_tl,
}
\makeatletter
\NewDocumentEnvironment{custom}{O{}}
{
\patchcmd{\@thm}{\trivlist}{\list{}{\leftmargin=\jessicaindent}}{}{}
\keys_set:nn { jessica/custom } { #1 }
\tl_set_eq:NN \jessicacustomname \l_jessica_custom_name_tl
\tl_set:Nx \l__jessica_custom_temp_tl
{
\bool_if:NTF \l_jessica_custom_numbered_bool
{ \exp_not:N \jessicacustom }
{ \exp_not:N \jessicacustomnonumber }
\tl_if_empty:NF \l_jessica_custom_attr_tl
{ [\exp_not:V \l_jessica_custom_attr_tl] }
}
\tl_use:N \l__jessica_custom_temp_tl
}
{
\bool_if:NTF \l_jessica_custom_numbered_bool
{ \endjessicacustom }
{ \endjessicacustomnonumber }
}
\makeatother
\ExplSyntaxOff
\begin{document}
\section{One}
\lipsum[4]
\begin{custom}[name=Custom Title,attr=optional words]\label{A}
\lipsum*[2]
\end{custom}
\begin{custom}\label{B}
Even though I've been teaching for n! years, every class is a fresh
adventure -- An adventure. That word is a euphemism for the stark reality,
which is that every class contains some totally shocking development
that I have never seen before and haven't any idea how to cope with.
Just last year, for the first time in many years, I taught a section
of the junior-level mathematical analysis course. You know -- the one
where students meet proofs in analysis for the first time. The one where
students and epsilons meet, eyeball to eyeball, and it isn't the epsilons
that blink. The one where students decide that they really wanted to be
doctors and lawyers after all.
\end{custom}
\begin{custom}[name=Notnumbered,nonumber,attr=xyz]
This is not numbered.
\end{custom}
Here are the references: \ref{A} and \ref{B}.
\end{document}