Nested enumerate (enumitem) interferes with suppressing vertical space after theorem heads.

Question

In the post How to suppress vertical space between theorem heads and enumitem environments? David Carlisle suggested a piece of code that helps suppressing vertical space between theorem heads and an enumerate environments. The code worked well, but I recently realized there is now additional vertical space inserted between enumerate environments when they are nested. How can this be avoided (without losing the feature of having vertical space suppressed after theorem heads)?

\documentclass{article}

\usepackage[T1]{fontenc}
\usepackage[american]{babel}
\usepackage{enumitem}
\usepackage{amsthm}

\newtheoremstyle{mythmstyle}%
{0.5em}% space above
{0.5em}% space below
{}% body font
{}% indent amount
{\sffamily\bfseries}% head font
{}% punctuation after head
{\newline}% space after head
{\thmname{#1}\ \thmnumber{#2}\ \thmnote{(#3)}}% head spec
\theoremstyle{mythmstyle}
\newtheorem{theorem}{Theorem}

% list settings
\setlist{% general list settings (enumitem's itemize, enumerate, and description)
  align=left,% left-aligned enumerate
  labelsep=*,% align all item bodies vertically
  leftmargin=*,% no left indent
  topsep=1mm,% space before enumerate
  itemsep=0mm% space between enumerate items
}
\setlist[enumerate,1]{label=\alph*)}% enumerate label on level 1
\setlist[enumerate,2]{label=\roman*)}% enumerate label on level 2

% vertical spacing after theorem heads (suggested by David Carlisle)
\makeatletter
\def\enumfix{%
\if@inlabel
 \noindent\par\nobreak\vskip-\parskip\vskip-\baselineskip\hrule\@height\z@
\fi}
\let\oldenumerate\enumerate
\def\enumerate{\enumfix\oldenumerate}
\makeatother

\begin{document}
\begin{theorem}% perfectly fine with David's solution: no vertical space
  \begin{enumerate}
  \item Foo
  \item Bar
  \end{enumerate}
\end{theorem}
\bigskip

\begin{enumerate}
\item Foobar
\item \begin{enumerate}% vertically not correctly aligned; comment out David's code above to see how it should look like
  \item Foo
  \item Bar
  \end{enumerate}
\end{enumerate}
\end{document}

Answer 1

I suggest you define a new environment for the first level enumerations in theorems, the only one that must be treated specially:

\documentclass{article}

\usepackage[T1]{fontenc}
\usepackage[american]{babel}
\usepackage{enumitem}
\usepackage{amsthm}

\newtheoremstyle{mythmstyle}%
{0.5em}% space above
{0.5em}% space below
{}% body font
{}% indent amount
{\sffamily\bfseries}% head font
{}% punctuation after head
{\newline}% space after head
{\thmname{#1}\ \thmnumber{#2}\ \thmnote{(#3)}}% head spec
\theoremstyle{mythmstyle}
\newtheorem{theorem}{Theorem}

% list settings
\setlist{% general list settings (enumitem's itemize, enumerate, and description)
  align=left,% left-aligned enumerate
  labelsep=*,% align all item bodies vertically
  leftmargin=*,% no left indent
  topsep=0mm,% space before enumerate
  itemsep=0mm% space between enumerate items
}
\setlist[enumerate,1]{label=\alph*)}% enumerate label on level 1
\setlist[enumerate,2]{label=\roman*)}% enumerate label on level 2

% vertical spacing after theorem heads (suggested by David Carlisle)
\makeatletter
\newenvironment{tenumerate}[1][]
 {\if@inlabel
    \noindent\par\nobreak\vskip-\parskip\vskip-\baselineskip\hrule\@height\z@
  \fi
  \enumerate[topsep=3pt,#1]}
 {\endenumerate}
\makeatother

\begin{document}
\begin{theorem}% perfectly fine with David's solution: no vertical space
  \begin{tenumerate}
  \item Foo
  \item Bar
  \end{tenumerate}
\end{theorem}
\bigskip

\begin{enumerate}
\item Foobar
\item \begin{enumerate}
  \item Foo
  \item Bar
  \end{enumerate}
\end{enumerate}
\end{document}

---

New way

It seems that a different strategy can work without a new environment:

\documentclass{article}

\usepackage[T1]{fontenc}
\usepackage[american]{babel}
\usepackage{enumitem}
\usepackage{amsthm}
\usepackage{etoolbox}

\newif\ifstarttheorem
\newtheoremstyle{mythmstyle}%
{0.5em}% space above
{0.5em}% space below
{}% body font
{}% indent amount
{\sffamily\bfseries\global\starttheoremtrue}% head font
{}% punctuation after head
{\newline}% space after head
{\thmname{#1}\ \thmnumber{#2}\ \thmnote{(#3)}}% head spec
\theoremstyle{mythmstyle}
\newtheorem{theorem}{Theorem}

% list settings
\setlist{% general list settings (enumitem's itemize, enumerate, and description)
  align=left,% left-aligned enumerate
  labelsep=*,% align all item bodies vertically
  leftmargin=*,% no left indent
  topsep=0mm,% space before enumerate
  itemsep=0mm% space between enumerate items
}
\setlist[enumerate,1]{label=\alph*)}% enumerate label on level 1
\setlist[enumerate,2]{label=\roman*)}% enumerate label on level 2

% vertical spacing after theorem heads (suggested by David Carlisle)
\makeatletter
\preto\enumerate{%
  \if@inlabel
    \ifstarttheorem
      \mbox{}\par\nobreak\vskip\glueexpr-\parskip-\baselineskip+3pt\relax\hrule\@height\z@
      \global\starttheoremfalse
    \fi
  \fi}
\preto\endenumerate{\global\starttheoremfalse}
\makeatother

\begin{document}

This is not a theorem.
\begin{enumerate}
\item Foo
\item Bar
\end{enumerate}

\begin{theorem}
This is a theorem.
\begin{enumerate}
\item Foo
\item Bar
\end{enumerate}
\end{theorem}

This is not a theorem.
\begin{enumerate}
\item Foo
\item Bar
\end{enumerate}

\begin{theorem}
  \begin{enumerate}
  \item Foo
  \item Bar
  \end{enumerate}
\end{theorem}


\begin{theorem}
\begin{enumerate}
\item Foobar
\item \begin{enumerate}
  \item Foo
  \item Bar
  \end{enumerate}
\end{enumerate}
\end{theorem}
\bigskip
\end{document}

Every theorem environment sets a conditional to true; if this conditional is true, an enumerate environment receives David's special treatment and sets the conditional to false. In order not to get false positives, also enumerate sets the conditional to false when it ends.