# Automatic indentation/framing of nested theorems/proofs

I'm a fan of python, and in the sadly common case where one creates new theorems, definitions, etc within the actual proofs of other theorems, I would like the "inner theorems" to be indented.

To this end, I've discovered this solution, which provides a frame as well as indenting it. However, that solution frames/indents everything, even base level theorems, which ended up being sort of ugly. Is there a way to make it frame/indent only when you're nesting it in another proof, short of defining separate environments and doing it manually?

Not sure if it's important, but in case it is, my current setup is:

\usepackage[amsmath,thmmarks,framed]{ntheorem}
\theorembodyfont{\small}
\usepackage{framed}
\usepackage{color}
\definecolor{gray}{rgb}{0.6,0.6,0.6}
\renewcommand*\FrameCommand{{\color{gray}\vrule width 5pt \hspace{10pt}}}

\theoremstyle{plain}
\newframedtheorem{thm}{Theorem}[page]
\newframedtheorem{lem}[thm]{Lemma}
\newframedtheorem{cor}[thm]{Corollary}
\newframedtheorem{prop}[thm]{Proposition}

\newframedtheorem{rem}[thm]{Remark}
\newframedtheorem{ques}[thm]{Question}

\theoremstyle{nonumberplain}
\newframedtheorem{proof}{Proof:}


Edit: my end solution was an lualatex modification of stuff posted in the checkmarked answer. Because he edited it out, I'll vaguely requote the general idea here:

Define two versions of your environments, thm and frthm, and within the proof definition, redefine \thm to \frthm and \endthm to endfrthm. This is not as nice a solution as the previous one, but it will technically work.

This is a MWE of that idea, written in lualatex for automation ( you'll have to modify the tex.print itself to change implementation-specific details like defining \newtheorem differently).

\documentclass[11pt,openany]{memoir}
\usepackage{cool}
\usepackage{luacode}
\usepackage{etoolbox}
\usepackage[amsmath,thmmarks,framed]{ntheorem}
\usepackage{framed}
\usepackage{color}
\definecolor{gray}{rgb}{0.6,0.6,0.6}

\begin{luacode}
prefix = "fr"

function createEnvironments(environs)
for sh, lo in pairs(environs) do
tex.print("\\newtheorem{" .. sh .. "}{" .. lo .. "}")
tex.print("\\newframedtheorem{" .. prefix .. sh .. "}{" .. lo .. "}")
end
end

function redefineEnvironments(environs)
for sh, lo in pairs(environs) do
tex.print("\\def\\" .. sh .. "{\\" ..  prefix .. sh .. "}" )
tex.print("\\def\\end" .. sh .. "{\\end" .. prefix .. sh .. "}")
end
end
\end{luacode}

\begin{luacode}
--your list of shortcut="Environment"; fill in this table!
environList = { lem="Lemma" , prop="Proposition",proof="Proof"}

createEnvironments(environList)
\end{luacode}

\renewcommand*\FrameCommand{{\color{gray}\vrule width 5pt \hspace{10pt}}}
\appto\proof{% ... within the proof environment, we redefine the commands to point to the framed environments
\directlua{redefineEnvironments(environList)}
}

\begin{document}
\begin{lem}
Layer 1
\end{lem}

\begin{proof}
Layer 0 stuff
\begin{prop}
Layer 1 starts getting embedded
\begin{lem}
This is an artefact: once you get into the first proof you start nesting no matter what
\end{lem}
\end{prop}

Layer 0 stuff
\begin{lem}
Another layer 1
\end{lem}

Can put layer 0 stuff here
\begin{proof}
Layer 1 proof
\begin{prop}
Layer 2
\end{prop}
\begin{proof}
We conclude with a layer 2 proof
\end{proof}
\end{proof}

\end{proof}
\end{document}


Explanation: I realized that making every theorem a framedtheorem had bad typesetting side-effects. This new version creates both and redefines the former to the latter within the proof environment, where nesting is both likely to occur. Of course, you can do the same within any other environment.

-
Can you add a full minimal working example that shows your solution completely. –  Marco Daniel Oct 4 '11 at 19:15
Sorry about that; done. –  scallops Oct 4 '11 at 20:42

Here's one approach, that uses etoolbox to tweak the definition of \FrameCommand from within the outermost theorem. A complete MWE follows:

\documentclass{article}
\usepackage[framed]{ntheorem}
\usepackage{framed} % To define \FrameCommand
\usepackage{color} % To define \definecolor
\usepackage{lipsum} % For the dummy text
\definecolor{gray}{rgb}{0.6,0.6,0.6}
\usepackage{etoolbox} % For \appto

\theoremstyle{plain}
\newframedtheorem{thm}{Theorem}[page]

\renewcommand*\FrameCommand{\relax} % By default, no framing occurs
\appto\thm{% ... but within the thm environment, FrameCommand is different
\renewcommand*\FrameCommand{{\color{gray}\vrule width 5pt \hspace{10pt}}}
}

\begin{document}
\noindent \lipsum[1]
\begin{thm}
Outer theorem
\begin{thm}
Theorem nested once
\begin{thm}
Theorem nested twice
\end{thm}
\end{thm}
\end{thm}
\end{document}


This results in the following appearance:

You'd need to add a similar call to \appto for each additional environment for which you wanted such nesting behavior. (Unfortunately, the original solution I proposed, using AtBeginEnvironment adds the hook too early in the environment definition, so the FrameCommand gets redefined before it's used in the outermost environment too. This version, using \appto\thm, is slightly less polite in that it hacks the opening of the environment by modifying the \thm command directly, but it adds the hook at the right moment.)

-
Hmm, the problem with the first solution seems to be that, \AtBeginEnvironment{proof}{\def\mynesting{\vrule width 5pt \hspace{10pt}}} automatically makes the proof environment always nested (even the first Proof and even when I redefined it back to \relax \AtEndEnvironment). I could be doing it wrong, because doing essentially the same thing via lua works (and lets me do more general stuff), so that's probably the approach I'll be taking. –  scallops Oct 4 '11 at 18:55
@scallops, See my edited answer: apparently (after testing it myself), AtBeginEnvironment takes effect earlier than I thought i did. There's still a very simple solution, using \appto instead, that gets the appearance I think you wanted. –  Ben Lerner Oct 4 '11 at 19:06
I like this new one as well. Very short and sweet. Thanks! –  scallops Oct 4 '11 at 19:17
@BenLerner: This solution has a big problem. You can't use frame breaks. If the content results in a frame break you will get en error. –  Marco Daniel Oct 4 '11 at 19:18
@Marco, I don't think this is the fault of my solution above, for two reasons: 1) if you comment out the \renewcommand and \appto commands, and add a bunch of \lipsum within the theorems, you'll see that the bare ntheorem styles yield either errors or an infinitely-long document. Also, 2) the change I made to \FrameCommand is in keeping with the definition of the leftbar environment in ntheorem, so I doubt that's the cause... I'm happy to edit the answer above to fix these bugs, but I don't know where to diagnose them :) Am I missing something obvious? –  Ben Lerner Oct 4 '11 at 19:35