# When a theorem is the first thing in a list, prevent extra space

I have an "indented" environment in my documents, which I create like this

\def\changemargin#1#2{
\list{}{\rightmargin#2\leftmargin#1}
\item[]
}
\let\endchangemargin=\endlist


Then if the first thing in the changemargin environment is a theorem, it will add an extra vertical space that shouldn't be there. How can I prevent this?

Examine

\documentclass[12pt]{article}
\usepackage{amsthm}
\usepackage[english]{babel}
\usepackage{blindtext}

\def\changemargin#1#2{\list{}{\rightmargin#2\leftmargin#1}\item[]}
\let\endchangemargin=\endlist

\newtheorem*{theorem}{Theorem}
\begin{document}

The rain in Spain falls mainly on the plain.
The rain in Spain falls mainly on the plain.
\begin{theorem}
This theorem has the correct spacing, because it's outside the changemargin environment.
\end{theorem}
The rain in Spain falls mainly on the plain. The rain in Spain falls mainly on the plain.

\begin{changemargin}{2cm}{2cm}
\begin{theorem}
This theorem has the wrong spacing, becuase it is the first thing in the change margin environment.
\end{theorem}
\end{changemargin}
The rain in Spain falls mainly on the plain. The rain in Spain falls mainly on the plain.
\begin{changemargin}{2cm}{2cm}
The rain in Spain falls mainly on the plain.
\begin{theorem}
This theorem has the correct spacing, because it is not the first thing in the changemargin environment.
\end{theorem}
\end{changemargin}
The rain in Spain falls mainly on the plain. The rain in Spain falls mainly on the plain.
\end{document}


One fix would be to create a macro for a theorem only to be used when it is the first thing in a changemargin environment. I'm wondering if there is anything better. Alternately, perhaps you could suggest a better way to change margins temporarily (not using list, I suppose) that wouldn't cause this problem.

Update: I am having trouble with the suggested answer below. A missing item error is being thrown to the following code.

\documentclass[12pt]{article}
\usepackage{amsthm}
\usepackage{xparse}
\makeatletter
\NewDocumentEnvironment{changemargin}{O{3cm}D<>{3cm}}%
{\list{}{\leftmargin=#1\rightmargin=#1}\item\@inlabelfalse\@newlistfalse}
{\endlist}
\makeatother

\newtheorem*{theorem}{Theorem}

\begin{document}
\begin{changemargin}[0.9cm]<0.5cm>
The rain in Spain falls mainly on the plain.
\end{changemargin}
\begin{theorem}
The rain in Spain falls mainly on the plain.
\end{theorem}
\end{document}

• With ntheorem you can create indented theorems, so you might begin the indented environment after the theorem. – Bernard Oct 2 '16 at 19:11
• @Bernard It appears to only increase the left margin, not the right margin. Is that right? – Eric Auld Oct 2 '16 at 19:17
• No. It is not really explained in the doc , but you also can change the right margin with \theoremrightindent some value (see lines 582-584 of the code, p.55 of the doc) – Bernard Oct 2 '16 at 19:39
• @Bernard That is helpful. To switch to ntheorem, I'd have to rewrite the definition of my theorem environment into the new syntax of ntheorem. Do you find ntheorem is worth the effort? – Eric Auld Oct 2 '16 at 19:56
• In my opinion it has an easier syntax than amsthm. Some problems have been reported in a few special cases, but I never met them. Its main advantage is its automatic placement of (customisable) end-of-proof symbol, even when the proof ends up in a displayed equation, and its ease of customisation. Finally, it can be used with thmtools. – Bernard Oct 2 '16 at 20:03

Two working answers has already been given to this questions; nonetheless, both answers fail to provide a detailed explanation of what is actually going on, and their authors plainly admit that their understanding of why those answers work is not perfect. Indeed, it is difficult—even for me—to tell whether or not the aforementiond solutions are guaranteed to work under any reasonable circumstance.

I decided to write this answer after getting a headache in the effort of sorting this out, essentially because it was clearly impossible to make it fit even in more than one comment. I hope that I’ll be able to shed some light on the matter, but please note that my understanding is not yet perfect too.

“Theorem-like” environments defined by means of the default facilities provided by the LaTeX kernel interact with lists in a perfectly correct way, as the following example shows:

% My standard header for TeX.SX answers:
\documentclass[a4paper]{article} % To avoid confusion, let us explicitly
% declare the paper format.

\usepackage[T1]{fontenc}         % Not always necessary, but recommended.
% End of standard header.  What follows pertains to the problem at hand.

% \usepackage{amsthm}

\newenvironment*{changemargins}[2]{%
\list{}{%
\setlength\leftmargin {#1}%
\setlength\rightmargin{#2}%
}\item\relax
}{\endlist}

\newtheorem{thrm}{Theorem}

\begin{document}

Theorem-like'' environments defined by means of the default facilities
provided by the \LaTeX\ kernel interact correctly with lists.

A theorem at the outer level:

\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}

Now inside a list: first making up an item by itself\ldots
\begin{enumerate}
\item
\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}

\item
\ldots and then after some text.
\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}

\item
A third item follows.
\end{enumerate}

In the same way, their behavior when used by themselves inside a
\texttt{changemargins} environment is absolutely normal:

\begin{changemargins}{2cm}{2cm}
\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}
\end{changemargins}

Another example, in which the theorem-like'' environment follows some text:

\begin{changemargins}{2cm}{2cm}
Some text, what suffices to get to the right margin\ldots\space
Well, perhaps just a little more.
\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}
\end{changemargins}

Uncommenting
\begin{verbatim}
\usepackage{amsthm}
\end{verbatim}
will break all the above examples, so it looks like the
\textsf{amsthm} package is the real culprit!

\end{document}


If you uncomment the \usepackage{amsthm} declaration, you will see that the “culprit” is actually this package. Indeed, the machinery of the various switches like \if@newlist, \if@inlabel, and so on, relies on the assumption that list-type environments are nested according to the following pattern:

\begin{list}{...}{...}
\item
\begin{list}{...}{...}
\item
Text


In other words, every list-type environment (including “trivlist-type” ones!) should directly enclose an \item command, or something equivalent, before any other actual contents, lest a ”missing \item” error be returned. But we can see (file amsthm.sty, lines 186–199) that the amsthm package, after having issued (line 123) a \trivlist, does not directly use an \item command, as the corresponding facility provided by the LaTeX kernel does, but rather “something equivalent”, which, though, does not honor the \if@noparitem switch.

The following example patches the relevant macro in the amsthm package, to make “theorem-like” environments behave as expected inside lists too. I prefer not to use the \patchcmd facility from the etoolbox package, both because the patch is a bit complex and for reasons of readability; bold guys are of course free to do differently.

% My standard header for TeX.SX answers:
\documentclass[a4paper]{article} % To avoid confusion, let us explicitly
% declare the paper format.

\usepackage[T1]{fontenc}         % Not always necessary, but recommended.
% End of standard header.  What follows pertains to the problem at hand.

\usepackage{amsthm}

\makeatletter

\if@noparlist
\@donoparitem
\else
\if@inlabel \indent \par \fi % eject a section head if one is pending
\if@nobreak
\else
\fi
\global\@inlabeltrue
\fi
\everypar\dth@everypar
\global\sbox\@labels{\unhbox\@labels\normalfont#1}%
\ignorespaces
}

\makeatother

\newenvironment*{changemargins}[2]{%
\list{}{%
\setlength\leftmargin {#1}%
\setlength\rightmargin{#2}%
}\item\relax
}{\endlist}

\newtheorem{thrm}{Theorem}

\begin{document}

Theorem-like'' environments defined by means of the default facilities
provided by the \LaTeX\ kernel interact correctly with lists.

A theorem at the outer level:

\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}

Now inside a list: first making up an item by itself\ldots
\begin{enumerate}
\item
\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}

\item
\ldots and then after some text.
\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}

\item
A third item follows.
\end{enumerate}

In the same way, their behavior when used by themselves inside a
\texttt{changemargins} environment is absolutely normal:

\begin{changemargins}{2cm}{2cm}
\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}
\end{changemargins}

Another example, in which the theorem-like'' environment follows some text:

\begin{changemargins}{2cm}{2cm}
Some text, what suffices to get to the right margin\ldots\space
Well, perhaps just a little more.
\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}
\end{changemargins}

\end{document}


Please note that, in order to keep the example simple, I’ve not considered the xparse syntax: that is an “orthogonal” feature.

This patch has the following advantages over the other ones:

• I know precisely why it works ;-) ;

• it works with all types of list environments.

Nevertheless, this does not mean that I am completely sure that it doesn’t break anything; indeed, the comment on line 187 of amsthm.sty suggests that it could. For this reason, I also propose another solution that keeps the patch to the amsthm macro local to the changemargins environment:

% My standard header for TeX.SX answers:
\documentclass[a4paper]{article} % To avoid confusion, let us explicitly
% declare the paper format.

\usepackage[T1]{fontenc}         % Not always necessary, but recommended.
% End of standard header.  What follows pertains to the problem at hand.

\usepackage{amsthm}

\makeatletter

\newcommand*\PatchAmsThmMacros{%
\if@noparlist
\@donoparitem
\else
\if@inlabel \indent \par \fi % eject a section head if one is pending
\if@nobreak
\else
\fi
\global\@inlabeltrue
\fi
\everypar\dth@everypar
\global\sbox\@labels{\unhbox\@labels\normalfont##1}%
\ignorespaces
}%
}

\makeatother

\newenvironment*{changemargins}[2]{%
\list{}{%
\setlength\leftmargin {#1}%
\setlength\rightmargin{#2}%
}%
\PatchAmsThmMacros
\item\relax
}{\endlist}

\newtheorem{thrm}{Theorem}

\begin{document}

Theorem-like'' environments defined by means of the default facilities
provided by the \LaTeX\ kernel interact correctly with lists.

A theorem at the outer level:

\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}

Now inside a list: first making up an item by itself\ldots
\begin{enumerate}
\item
\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}

\item
\ldots and then after some text.
\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}

\item
A third item follows.
\end{enumerate}

In the same way, their behavior when used by themselves inside a
\texttt{changemargins} environment is absolutely normal:

\begin{changemargins}{2cm}{2cm}
\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}
\end{changemargins}

Another example, in which the theorem-like'' environment follows some text:

\begin{changemargins}{2cm}{2cm}
Some text, what suffices to get to the right margin\ldots\space
Well, perhaps just a little more.
\begin{thrm}
The ducks in dutch quack quaveringly on the docks.
\end{thrm}
\end{changemargins}

\end{document}


As you can see, this makes the patch work with the changemargins environment, but not with enumerate.

• Thank you very much — this fixed an outstanding problem for my users. – Peter Flynn Mar 26 '18 at 7:42

Note: I don't know what I'm doing.

# Caveat emptor ...

I added the additions just to check I'd not broken the spacing inadvertently. The code change is from latex.ltx. Also, I corrected the use of #1 in place of #2.

\documentclass[12pt]{article}
\usepackage{amsthm}
\usepackage{xparse}
\makeatletter
\NewDocumentEnvironment{changemargin} {O{3cm}D<>{3cm}}
{%
\list{}{\leftmargin=#1\rightmargin=#2}\item
\if@inlabel\global\@inlabelfalse\fi
\@newlistfalse
}{%
\endlist
}
\makeatother

\newtheorem*{theorem}{Theorem}

\begin{document}
\begin{changemargin}[0.9cm]<0.5cm>
The rain in Spain falls mainly on the plain.
\begin{theorem}
The rain in Spain falls mainly on the plain.
\end{theorem}
\end{changemargin}
The rain in Spain falls mainly on the plain.
\begin{theorem}
The rain in Spain falls mainly on the plain.
\end{theorem}
\end{document}


• Using \global might have unintended effects with nesting...but perhaps not. What is going wrong with the OP's example is that the flags are screwed up for the next theorem environment, which is outside the changemargin environment. I'll update my solution with what is probably a safer solution... Thanks for fixing my #2:) – Andrew Oct 8 '16 at 11:58
• @Andrew You are probably right. I just looked at similar cases in latex.ltx to see what they did when using \if@inlabel etc. I did check that the flag is set globally to false by the format. – cfr Oct 8 '16 at 14:24

EDIT (rewrite and simplification of code)

I delved back into the definition of theorem-like environments and I think that I found what causes the problem when trying to use \list to do this. Namely, it is lines 4559-4566 from latex.ltx which form part of the definition of \trivlist:

\if@inlabel
\@noparitemtrue
\@noparlisttrue
\else
\if@newlist \@noitemerr \fi
\@noparlistfalse
\fi


What is happening is that \list sets \@inlabeltrue and, in turn, this leads to extra space being added at the top of any environment that uses \trivlist, of which there are many. To stop this happening we just need to disable this. Once this is done the MWE in the question produces:

Here is the new code, which is quite a bit simpler than my first attempt:

\documentclass[12pt]{article}
\usepackage{amsthm}
\usepackage[margin=16mm,showframe]{geometry}% to show the margins

\usepackage{xparse}
\makeatletter
\NewDocumentEnvironment{changemargin}{O{3cm}D<>{3cm}}%
{\list{}{\leftmargin=#1\rightmargin=#2}\item%
\bgroup\@inlabelfalse\@newlistfalse}% change flags only in a group
{\egroup\endlist}
\makeatother

\newtheorem*{theorem}{Theorem}

\begin{document}

The rain in Spain falls mainly on the plain.
The rain in Spain falls mainly on the plain.
\begin{theorem}
This theorem has the correct spacing, because it's outside the changemargin environment.
\end{theorem}
The rain in Spain falls mainly on the plain. The rain in Spain falls mainly on the plain.

\begin{changemargin}[2cm]<2cm>
\begin{theorem}
This theorem has the right spacing and it is the first thing in the change margin environment.
\end{theorem}
\end{changemargin}
The rain in Spain falls mainly on the plain. The rain in Spain falls mainly on the plain.
\begin{changemargin}[2cm]<2cm>
The rain in Spain falls mainly on the plain.
\begin{theorem}
This theorem has the correct spacing, because it is not the first thing in the changemargin environment.
\end{theorem}
\end{changemargin}
The rain in Spain falls mainly on the plain. The rain in Spain falls mainly on the plain.

\end{document}


The \@newlistfalse is necessary because without latex gives an error, saying that there is a missing \item. As far as I can see there are no additional side-effects of adding the lines \@inlabelfalse\@newlistfalse as they will simply stop extra space being added to any environment that uses \trivlist. Of course, I may have missed something!

The only other comment worth making is that I have changed the OP's \changemargin macro into an enviroment by using \NewDocumentEnvironment from xparse. The environment has two optional arguments, which can be used independently, for setting the margins in the changemargin environment. For example,

  \begin{changemargin}[2cm]<4cm>
Here is the correct spacing.
\end{changemargin}


sets the left-hand margin to 2cm and the right-hand margin to 4cm. Both margins default to 3mm.

• @EricAuld I have replaced my hack with a solution. I will delete my other comments here as they are no longer relevant. – Andrew Oct 7 '16 at 5:07
• There's a "missing item" error being thrown. Check out the code in the question. Perhaps you can help me figure out what's wrong. – Eric Auld Oct 7 '16 at 19:00
• @EricAuld What is happening in your example is that the @inlabel flag is wrong for the next theorem environment when it's not inside a changemargin environment. This was definitely a mistake in my code as the changemargin environment should not change the "state" of external environments. I think that the best way to fix is to make these changes "local" by putting them inside a group. I have updated my answer so that it does this. – Andrew Oct 8 '16 at 12:01
• I’m afraid your analysis is not correct: try \begin{enumerate}\item \begin{trivlist}\item Foo.\item Bar.\end{trivlist} \end{enumerate}, and you’ll see that it is handled correctly. The point is that the “theorem-like” environments defined by the amsthm package, albeit implemented through a trvilist environment, do not honor \@noparitemtrue (see amsthm.sty, lines 186–199). – GuM Oct 8 '16 at 20:08