Odd custom theorem behavior when trying to set default value of optional argument

Question

MWE:

\documentclass{book}
\usepackage{amsthm}
\usepackage{enumitem}
\usepackage{xparse}
\usepackage{hyperref}

\usepackage{lipsum}

\newtheoremstyle{fctaylor}% name
  {\topsep}%      Space above
  {\topsep}%      Space below
  {\normalfont}%         Body font
  {}%         Indent amount (empty = no indent, \parindent = para indent)
  {\bfseries}% Thm head font
  {}%        Punctuation after thm head
  {0pt}%     Space after thm head: " " = normal interword space;
  {\makethmhead{#1}{#2}{#3}}

\newlength\fctaylortheoremindent
\AtBeginDocument{\setlength\fctaylortheoremindent{3em}} % <- customize here
\newlength\fctaylorlabelsep
\AtBeginDocument{\setlength\fctaylorlabelsep{1em}} % <- customize here

\makeatletter
\newcommand{\makethmhead}[3]{%
  \gdef\thisthmhead{%
    \makebox[\fctaylortheoremindent][l]{\bfseries#2}%
    {\bfseries#1}%
    \@ifnotempty{#3}{ (#3)}%
    \hspace{\fctaylorlabelsep}%
    \phantomsection%%% THIS LINE ADDED
  }%
}
\makeatother

\newenvironment{fctayloritemize}
 {\list{}{%
    \leftmargin=\fctaylortheoremindent
    \labelwidth=\dimexpr\fctaylortheoremindent-\labelsep\relax
    \itemindent=0pt
  }}
 {\endlist}

\NewDocumentCommand{\newfctaylortheorem}{smomo}{%
  \IfBooleanTF{#1}
   {\newtheorem*{fctaylor@#2}{#4}}
   {\IfNoValueTF{#3}
     {\IfNoValueTF{#5}
       {\newtheorem{fctaylor@#2}{#4}}
       {\newtheorem{fctaylor@#2}{#4}[#5]}}
     {\newtheorem{fctaylor@#2}[fctaylor@#3]{#4}}}%
  \NewDocumentEnvironment{#2}{o}
   {\IfNoValueTF{##1}{\begin{fctaylor@#2}}{\begin{fctaylor@#2}[##1]}%
    \begin{fctayloritemize}\item[\thisthmhead\hfill]}
   {\end{fctayloritemize}\end{fctaylor@#2}}%
}

\theoremstyle{fctaylor}
\newfctaylortheorem{mytheorem}{Theorem}[chapter]
\newcommand{\thistheoremname}{}
\newfctaylortheorem{genericthm}[mytheorem]{\thistheoremname}
\newenvironment{custom}[1][Custom]
  {\renewcommand{\thistheoremname}{#1}%
   \begin{genericthm}}
  {\end{genericthm}}
\newfctaylortheorem*{genericthm*}{\thistheoremname}
\newenvironment{custom*}[1][Custom]
  {\renewcommand{\thistheoremname}{#1}%
   \begin{genericthm*}}
  {\end{genericthm*}}

\begin{document}
\section{One}

\begin{custom}{Custom Title}[optional words]\label{A}
\lipsum*[2]
\end{custom}

\newpage

\begin{custom}\label{B}
Even though I've been teaching for n! years, every class is a fresh
adventure -- An adventure. That word is a euphemism for the stark reality,
which is that every class contains some totally shocking development
that I have never seen before and haven't any idea how to cope with.

Just last year, for the first time in many years, I taught a section
of the junior-level mathematical analysis course. You know --  the one
where students meet proofs in analysis for the first time. The one where
students and epsilons meet, eyeball to eyeball, and it isn't the epsilons
that blink. The one where students decide that they really wanted to be
doctors and lawyers after all.
\end{custom}

\newpage

Here are the references: \ref{A} and \ref{B}.

\end{document}

I am trying to mimic this answer by making it possible to have a custom theorem title and optional argument, but I want to set the default value of the optional argument. When doing so, I get the following:

If I attempt to remove the default value, then I get the error ! Argument of \label has an extra }. If I do not use a label, then I get this:

Is there an easy way to fix this issue? Possibly unrelated, but I heard I could more easily put numbers on the left by using \swapnumbers but does that have any bearing here on the behavior I am trying to eradicate? (Note: I got my theorem environments from this answer.)

Desired/expected output:

\begin{custom}{Custom Title}[optional words]\label{A}
\lipsum*[2]
\end{custom}

Would produce

and

\begin{custom}\label{B}
...
\end{custom}

would produce

Answer 1

You might use the G specifier, which denotes an optional argument in braces:

\documentclass{article}
\usepackage{amsthm,xparse}

\usepackage{lipsum}

\makeatletter
\newtheoremstyle{jessica}% name
  {\topsep}%      Space above
  {\topsep}%      Space below
  {\normalfont}%         Body font
  {}%         Indent amount (empty = no indent, \parindent = para indent)
  {\bfseries}% Thm head font
  {}%        Punctuation after thm head
  {0pt}%     Space after thm head: " " = normal interword space;
  {%
   \makebox[0pt][r]{\makebox[\jessicaindent][l]{#2}}%
   #1%
   \@ifnotempty{#3}{\textnormal{ (#3)}}%    
   \hspace{\jessicalabelsep}%
  }
\makeatother

\newlength{\jessicaindent}
\newlength{\jessicalabelsep}
\setlength{\jessicaindent}{4em}
\setlength{\jessicalabelsep}{0.5em}

\theoremstyle{jessica}
\newtheorem{jessicacustom}{\jessicacustomname}[section]
\newtheorem*{jessicacustomnonumber}{\jessicacustomname}

\NewDocumentEnvironment{custom}{G{Custom}}
 {\def\jessicacustomname{#1}\jessicacustom}
 {\endjessicacustom}

\NewDocumentEnvironment{custom*}{G{Custom}}
 {\def\jessicacustomname{#1}\jessicacustomnonumber}
 {\endjessicacustomnonumber}

\begin{document}

\section{One}

\lipsum[4]

\begin{custom}{Custom Title}[optional words]\label{A}
\lipsum*[2]
\end{custom}

\begin{custom}\label{B}
Even though I've been teaching for n! years, every class is a fresh
adventure -- An adventure. That word is a euphemism for the stark reality,
which is that every class contains some totally shocking development
that I have never seen before and haven't any idea how to cope with.

Just last year, for the first time in many years, I taught a section
of the junior-level mathematical analysis course. You know --  the one
where students meet proofs in analysis for the first time. The one where
students and epsilons meet, eyeball to eyeball, and it isn't the epsilons
that blink. The one where students decide that they really wanted to be
doctors and lawyers after all.
\end{custom}

\begin{custom*}{Notnumbered}[xyz]
This is not numbered.
\end{custom*}

Here are the references: \ref{A} and \ref{B}.

\end{document}

I'd prefer a key-value interface:

\documentclass{article}
\usepackage{amsthm,xparse}

\usepackage{lipsum}

\makeatletter
\newtheoremstyle{jessica}% name
  {\topsep}%      Space above
  {\topsep}%      Space below
  {\normalfont}%         Body font
  {}%         Indent amount (empty = no indent, \parindent = para indent)
  {\bfseries}% Thm head font
  {}%        Punctuation after thm head
  {0pt}%     Space after thm head: " " = normal interword space;
  {%
   \makebox[0pt][r]{\makebox[\jessicaindent][l]{#2}}%
   #1%
   \@ifnotempty{#3}{\textnormal{ (#3)}}%    
   \hspace{\jessicalabelsep}%
  }
\makeatother

\newlength{\jessicaindent}
\newlength{\jessicalabelsep}
\setlength{\jessicaindent}{4em}
\setlength{\jessicalabelsep}{0.5em}

\theoremstyle{jessica}
\newtheorem{jessicacustom}{\jessicacustomname}[section]
\newtheorem*{jessicacustomnonumber}{\jessicacustomname}

\ExplSyntaxOn
\keys_define:nn { jessica/custom }
 {
  number   .bool_set:N         = \l_jessica_custom_numbered_bool,
  number   .default:n          = true,
  number   .initial:n          = true,
  nonumber .bool_set_inverse:N = \l_jessica_custom_numbered_bool,
  nonumber .default:n          = true,
  name     .tl_set:N           = \l_jessica_custom_name_tl,
  name     .initial:n          = Custom,
  attr .tl_set:N = \l_jessica_custom_attr_tl,
 }
\tl_new:N \l__jessica_custom_temp_tl

\NewDocumentEnvironment{custom}{O{}}
 {
  \keys_set:nn { jessica/custom } { #1 }
  \tl_set_eq:NN \jessicacustomname \l_jessica_custom_name_tl
  \tl_set:Nx \l__jessica_custom_temp_tl
   {
    \bool_if:NTF \l_jessica_custom_numbered_bool
     { \exp_not:N \jessicacustom }
     { \exp_not:N \jessicacustomnonumber }
    \tl_if_empty:NF \l_jessica_custom_attr_tl
     { [\exp_not:V \l_jessica_custom_attr_tl] }
   }
  \tl_use:N \l__jessica_custom_temp_tl
 }
 {
  \bool_if:NTF \l_jessica_custom_numbered_bool
   { \endjessicacustom }
   { \endjessicacustomnonumber }
 }
\ExplSyntaxOff

\begin{document}

\section{One}

\lipsum[4]

\begin{custom}[name=Custom Title,attr=optional words]\label{A}
\lipsum*[2]
\end{custom}

\begin{custom}\label{B}
Even though I've been teaching for n! years, every class is a fresh
adventure -- An adventure. That word is a euphemism for the stark reality,
which is that every class contains some totally shocking development
that I have never seen before and haven't any idea how to cope with.

Just last year, for the first time in many years, I taught a section
of the junior-level mathematical analysis course. You know --  the one
where students meet proofs in analysis for the first time. The one where
students and epsilons meet, eyeball to eyeball, and it isn't the epsilons
that blink. The one where students decide that they really wanted to be
doctors and lawyers after all.
\end{custom}

\begin{custom}[name=Notnumbered,nonumber,attr=xyz]
This is not numbered.
\end{custom}

Here are the references: \ref{A} and \ref{B}.

\end{document}

---

Version with the statement indented, with code adapted from https://tex.stackexchange.com/a/236407/4427

\documentclass{article}
\usepackage{amsthm,xparse,etoolbox}

\usepackage{lipsum}

\makeatletter
\newlength{\jessicaindent}
\newlength{\jessicalabelsep}
\setlength{\jessicaindent}{4em}
\setlength{\jessicalabelsep}{0.5em}

\newtheoremstyle{jessica}% name
  {\topsep}%      Space above
  {\topsep}%      Space below
  {\normalfont}%         Body font
  {}%         Indent amount (empty = no indent, \parindent = para indent)
  {\bfseries}% Thm head font
  {}%        Punctuation after thm head
  {0pt}%     Space after thm head: " " = normal interword space;
  {%
   \makebox[0pt][r]{\makebox[\jessicaindent][l]{#2}}%
   #1%
   \@ifnotempty{#3}{\textnormal{ (#3)}}%    
   \hspace{\jessicalabelsep}%
  }

\theoremstyle{jessica}
\newtheorem{jessicacustom}{\jessicacustomname}[section]
\newtheorem*{jessicacustomnonumber}{\jessicacustomname}

\NewDocumentEnvironment{custom}{G{Custom}}
 {%
  \patchcmd{\@thm}{\trivlist}{\list{}{\leftmargin=\jessicaindent}}{}{}%
  \def\jessicacustomname{#1}\jessicacustom
 }
 {\endjessicacustom}
\patchcmd{\endjessicacustom}{\endtrivlist}{\endlist}{}{}

\NewDocumentEnvironment{custom*}{G{Custom}}
 {%
  \patchcmd{\@thm}{\trivlist}{\list{}{\leftmargin=\jessicaindent}}{}{}%
  \def\jessicacustomname{#1}\jessicacustomnonumber
 }
 {\endjessicacustomnonumber}
\patchcmd{\endjessicacustomnonumber}{\endtrivlist}{\endlist}{}{}
\makeatother

\begin{document}

\section{One}

\lipsum[4]

\begin{custom}{Custom Title}[optional words]\label{A}
\lipsum*[2]
\end{custom}

\begin{custom}\label{B}
Even though I've been teaching for n! years, every class is a fresh
adventure -- An adventure. That word is a euphemism for the stark reality,
which is that every class contains some totally shocking development
that I have never seen before and haven't any idea how to cope with.

Just last year, for the first time in many years, I taught a section
of the junior-level mathematical analysis course. You know --  the one
where students meet proofs in analysis for the first time. The one where
students and epsilons meet, eyeball to eyeball, and it isn't the epsilons
that blink. The one where students decide that they really wanted to be
doctors and lawyers after all.
\end{custom}

\begin{custom*}{Notnumbered}[xyz]
This is not numbered.
\end{custom*}

Here are the references: \ref{A} and \ref{B}.

\end{document}

The same with the key-value interface.

\documentclass{article}
\usepackage{amsthm,xparse,etoolbox}

\usepackage{lipsum}

\makeatletter
\newlength{\jessicaindent}
\newlength{\jessicalabelsep}
\setlength{\jessicaindent}{4em}
\setlength{\jessicalabelsep}{0.5em}

\newtheoremstyle{jessica}% name
  {\topsep}%      Space above
  {\topsep}%      Space below
  {\normalfont}%         Body font
  {}%         Indent amount (empty = no indent, \parindent = para indent)
  {\bfseries}% Thm head font
  {}%        Punctuation after thm head
  {0pt}%     Space after thm head: " " = normal interword space;
  {%
   \makebox[0pt][r]{\makebox[\jessicaindent][l]{#2}}%
   #1%
   \@ifnotempty{#3}{\textnormal{ (#3)}}%    
   \hspace{\jessicalabelsep}%
  }

\theoremstyle{jessica}
\newtheorem{jessicacustom}{\jessicacustomname}[section]
\newtheorem*{jessicacustomnonumber}{\jessicacustomname}
\patchcmd{\endjessicacustom}{\endtrivlist}{\endlist}
\patchcmd{\endjessicacustomnonumber}{\endtrivlist}{\endlist}

\makeatother

\ExplSyntaxOn
\keys_define:nn { jessica/custom }
 {
  number   .bool_set:N         = \l_jessica_custom_numbered_bool,
  number   .default:n          = true,
  number   .initial:n          = true,
  nonumber .bool_set_inverse:N = \l_jessica_custom_numbered_bool,
  nonumber .default:n          = true,
  name     .tl_set:N           = \l_jessica_custom_name_tl,
  name     .initial:n          = Custom,
  attr .tl_set:N = \l_jessica_custom_attr_tl,
 }

\makeatletter
\NewDocumentEnvironment{custom}{O{}}
 {
  \patchcmd{\@thm}{\trivlist}{\list{}{\leftmargin=\jessicaindent}}{}{}
  \keys_set:nn { jessica/custom } { #1 }
  \tl_set_eq:NN \jessicacustomname \l_jessica_custom_name_tl
  \tl_set:Nx \l__jessica_custom_temp_tl
   {
    \bool_if:NTF \l_jessica_custom_numbered_bool
     { \exp_not:N \jessicacustom }
     { \exp_not:N \jessicacustomnonumber }
    \tl_if_empty:NF \l_jessica_custom_attr_tl
     { [\exp_not:V \l_jessica_custom_attr_tl] }
   }
  \tl_use:N \l__jessica_custom_temp_tl
 }
 {
  \bool_if:NTF \l_jessica_custom_numbered_bool
   { \endjessicacustom }
   { \endjessicacustomnonumber }
 }
\makeatother
\ExplSyntaxOff

\begin{document}

\section{One}

\lipsum[4]

\begin{custom}[name=Custom Title,attr=optional words]\label{A}
\lipsum*[2]
\end{custom}

\begin{custom}\label{B}
Even though I've been teaching for n! years, every class is a fresh
adventure -- An adventure. That word is a euphemism for the stark reality,
which is that every class contains some totally shocking development
that I have never seen before and haven't any idea how to cope with.

Just last year, for the first time in many years, I taught a section
of the junior-level mathematical analysis course. You know --  the one
where students meet proofs in analysis for the first time. The one where
students and epsilons meet, eyeball to eyeball, and it isn't the epsilons
that blink. The one where students decide that they really wanted to be
doctors and lawyers after all.
\end{custom}

\begin{custom}[name=Notnumbered,nonumber,attr=xyz]
This is not numbered.
\end{custom}

Here are the references: \ref{A} and \ref{B}.

\end{document}

Answer 2

You wish to do both

\begin{custom}

and

\begin{custom}{Custom Title}[optional words]

. Although the first argument of the custom-environment, the phrase {Custom Title}, is nested in curly braces, it actually is intended to be an optional argument as well.

On the one hand I doubt that overloading the curly-brace-syntax, which usually is for denoting mandatory arguments, by also using it for denoting optional arguments is a good idea.

On the other hand you cannot have consecutive optional arguments with square-brackets: If you don't deliver all of them, it is unclear which ones you wish to provide and which ones you wish to omit in order to have their default-values.

You were already suggested using some keyval-interface.
I think that is the best approach to the matter.

Be that as it may:

For the sake of having fun I implemented a crude hack for intercepting \csname..\endcsname-evaluation of environment-names:

You have defined an environment whose name is custom and another environment whose name is custom*.

Instead of providing to the \begin-macro one of these <environment-names> as argument directly, you can provide a call to the macro \Customtitle:

Syntax:

\Customtitle[<Custom title>]{<environment-name>}
 ; <environment-name>  = custom  or  custom*

Putting the pieces together:

First example:

  \begin{custom}
  ...
  \end{custom}

Second example:

  \begin{custom}[optional words]
  ...
  \end{custom}

Third example:

  \begin{\Customtitle[<Custom title>]{custom}}
  ...
  \end{custom}

Fourth example:

  \begin{\Customtitle[<Custom title>]{custom}}[optional words]
  ...
  \end{custom}

Fifth example - this is equivalent to the first example:

  \begin{\Customtitle{custom}}
  ...
  \end{custom}

Sixth example - this is equivalent to the second example:

  \begin{\Customtitle{custom}}[optional words]
  ...
  \end{custom}

-----------------------------------------------------------

Seventh example:

  \begin{custom*}
  ...
  \end{custom*}

Eighth example:

  \begin{custom*}[optional words]
  ...
  \end{custom*}

Nineth example:

  \begin{\Customtitle[<Custom title>]{custom*}}
  ...
  \end{custom*}

Tenth example:

  \begin{\Customtitle[<Custom title>]{custom*}}[optional words]
  ...
  \end{custom*}

Eleventh example - this is equivalent to the seventh example:

  \begin{\Customtitle{custom*}}
  ...
  \end{custom*}

Twelveth example - this is equivalent to the eighth example:

  \begin{\Customtitle{custom*}}[optional words]
  ...
  \end{custom*}

 

\documentclass{book}
\usepackage{amsthm}
\usepackage{enumitem}
\usepackage{xparse}
\usepackage{hyperref}

\usepackage{lipsum}

\newtheoremstyle{fctaylor}% name
  {\topsep}%      Space above
  {\topsep}%      Space below
  {\normalfont}%         Body font
  {}%         Indent amount (empty = no indent, \parindent = para indent)
  {\bfseries}% Thm head font
  {}%        Punctuation after thm head
  {0pt}%     Space after thm head: " " = normal interword space;
  {\makethmhead{#1}{#2}{#3}}

\newlength\fctaylortheoremindent
\AtBeginDocument{\setlength\fctaylortheoremindent{3em}} % <- customize here
\newlength\fctaylorlabelsep
\AtBeginDocument{\setlength\fctaylorlabelsep{1em}} % <- customize here

\makeatletter
\newcommand{\makethmhead}[3]{%
  \gdef\thisthmhead{%
    \makebox[\fctaylortheoremindent][l]{\bfseries#2}%
    {\bfseries#1}%
    \@ifnotempty{#3}{ (#3)}%
    \hspace{\fctaylorlabelsep}%
    \phantomsection%%% THIS LINE ADDED
  }%
}
\makeatother

\newenvironment{fctayloritemize}
 {\list{}{%
    \leftmargin=\fctaylortheoremindent
    \labelwidth=\dimexpr\fctaylortheoremindent-\labelsep\relax
    \itemindent=0pt
  }}
 {\endlist}

\NewDocumentCommand{\newfctaylortheorem}{smomo}{%
  \IfBooleanTF{#1}
   {\newtheorem*{fctaylor@#2}{#4}}
   {\IfNoValueTF{#3}
     {\IfNoValueTF{#5}
       {\newtheorem{fctaylor@#2}{#4}}
       {\newtheorem{fctaylor@#2}{#4}[#5]}}
     {\newtheorem{fctaylor@#2}[fctaylor@#3]{#4}}}%
  \NewDocumentEnvironment{#2}{o}
   {\IfNoValueTF{##1}{\begin{fctaylor@#2}}{\begin{fctaylor@#2}[##1]}%
    \begin{fctayloritemize}\item[\thisthmhead\hfill]}
   {\end{fctayloritemize}\end{fctaylor@#2}}%
}

\theoremstyle{fctaylor}
\newfctaylortheorem{mytheorem}{Theorem}[chapter]
\newcommand{\thistheoremname}{}
\newfctaylortheorem{genericthm}[mytheorem]{\thistheoremname}

\makeatletter
\newcommand\Customtitle{@Customtitle\endcsname}
\newcommand\@Customtitle[2][Custom]{%
  \@ifundefined{#2}{%
    \def\reserved@a{\@latex@error{Environment #2 undefined}\@eha}%
  }{%
    \def\reserved@a{%
      \def\@currenvir{#2}%
      \renewcommand{\thistheoremname}{#1}%
      \expandafter\expandafter\expandafter\@gobbletwo\csname#2\endcsname
    }%
  }%
  \csname reserved@a%
}%
\newenvironment{custom}%
  {\@firstofone{\renewcommand{\thistheoremname}{Custom}}%
   \begin{genericthm}}
  {\end{genericthm}}
\newfctaylortheorem*{genericthm*}{\thistheoremname}
\newenvironment{custom*}%
  {\@firstofone{\renewcommand{\thistheoremname}{Custom}}%
   \begin{genericthm*}}
  {\end{genericthm*}}
\makeatother

\begin{document}
\section{One}

\begin{custom}\label{A}
\lipsum*[2]
\end{custom}

\begin{\Customtitle{custom}}\label{B}
\lipsum*[2]
\end{custom}

\begin{custom}[optional words]\label{C}
\lipsum*[2]
\end{custom}

\begin{\Customtitle{custom}}[optional words]\label{D}
\lipsum*[2]
\end{custom}

\begin{\Customtitle[Custom Title]{custom}}\label{E}
\lipsum*[2]
\end{custom}

\begin{\Customtitle[Custom Title]{custom}}[optional words]\label{F}
\lipsum*[2]
\end{custom}

\newpage

\begin{custom*}\label{G}
\lipsum*[2]
\end{custom*}

\begin{\Customtitle{custom*}}\label{H}
\lipsum*[2]
\end{custom*}

\begin{custom*}[optional words]\label{I}
\lipsum*[2]
\end{custom*}

\begin{\Customtitle{custom*}}[optional words]\label{J}
\lipsum*[2]
\end{custom*}

\begin{\Customtitle[Custom Title]{custom*}}\label{K}
\lipsum*[2]
\end{custom*}

\begin{\Customtitle[Custom Title]{custom*}}[optional words]\label{L}
\lipsum*[2]
\end{custom*}

\newpage

Here are the references:\bigskip

Here the hyperlinks go to the hyperlink-anchors created by the 
theorems via \verb|\phantompage| while textual phrases denote 
the numbers of the theorems. As the counter of \verb|custom|-theorems
is bound do the chapter counter, the theorem-numbers are of pattern
\verb|<chapter>.<theorem>|. With \verb|<chapter>=0| you get the
pattern \verb|0.<theorem>|:\bigskip

\ref{A} and \ref{B} and \ref{C} and \ref{D} and \ref{E}
and \ref{F}.\bigskip

Here the hyperlinks go to the hyperlink-anchors created by the
theorems via \verb|\phantompage| while textual phrases denote
the number of the section in question as \verb|custom*|-theorems
don't have a counter. We are in chapter 0, section 1, thus the
section number is 0.1, thus you always get \verb|0.1|:\bigskip

\ref{G} and \ref{H} and \ref{I} and \ref{J} and \ref{K}
and \ref{L}.

\end{document}