Automatic indentation/framing of nested theorems/proofs

Question

I'm a fan of python, and in the sadly common case where one creates new theorems, definitions, etc within the actual proofs of other theorems, I would like the "inner theorems" to be indented.

To this end, I've discovered this solution, which provides a frame as well as indenting it. However, that solution frames/indents everything, even base level theorems, which ended up being sort of ugly. Is there a way to make it frame/indent only when you're nesting it in another proof, short of defining separate environments and doing it manually?

Not sure if it's important, but in case it is, my current setup is:

\usepackage[amsmath,thmmarks,framed]{ntheorem}
\theorembodyfont{\small}
\usepackage{framed}
\usepackage{color}
\definecolor{gray}{rgb}{0.6,0.6,0.6}
\renewcommand*\FrameCommand{{\color{gray}\vrule width 5pt \hspace{10pt}}}

\theoremstyle{plain}
\newframedtheorem{thm}{Theorem}[page]
\newframedtheorem{lem}[thm]{Lemma}
\newframedtheorem{cor}[thm]{Corollary}
\newframedtheorem{prop}[thm]{Proposition}

\theoremheaderfont{\normalfont\itshape}
\newframedtheorem{rem}[thm]{Remark}
\newframedtheorem{ques}[thm]{Question}

\theoremstyle{nonumberplain}
\theoremheaderfont{\normalfont\itshape}
\newframedtheorem{proof}{Proof:}

---

Edit: my end solution was an lualatex modification of stuff posted in the checkmarked answer. Because he edited it out, I'll vaguely requote the general idea here:

Define two versions of your environments, thm and frthm, and within the proof definition, redefine \thm to \frthm and \endthm to endfrthm. This is not as nice a solution as the previous one, but it will technically work.

This is a MWE of that idea, written in lualatex for automation ( you'll have to modify the tex.print itself to change implementation-specific details like defining \newtheorem differently).

\documentclass[11pt,openany]{memoir}
\usepackage{cool}
\usepackage{luacode}
\usepackage{etoolbox}
\usepackage[amsmath,thmmarks,framed]{ntheorem}
\usepackage{framed}
\usepackage{color}
\definecolor{gray}{rgb}{0.6,0.6,0.6}

\begin{luacode}
prefix = "fr"

function createEnvironments(environs)
    for sh, lo in pairs(environs) do
        tex.print("\\newtheorem{" .. sh .. "}{" .. lo .. "}")
        tex.print("\\newframedtheorem{" .. prefix .. sh .. "}{" .. lo .. "}")
    end
end

function redefineEnvironments(environs)
    for sh, lo in pairs(environs) do
        tex.print("\\def\\" .. sh .. "{\\" ..  prefix .. sh .. "}" )
        tex.print("\\def\\end" .. sh .. "{\\end" .. prefix .. sh .. "}")
    end
end
\end{luacode}

\begin{luacode}
--your list of shortcut="Environment"; fill in this table!
environList = { lem="Lemma" , prop="Proposition",proof="Proof"}

createEnvironments(environList)
\end{luacode}

\renewcommand*\FrameCommand{{\color{gray}\vrule width 5pt \hspace{10pt}}}
\appto\proof{% ... within the proof environment, we redefine the commands to point to the framed environments
\directlua{redefineEnvironments(environList)}
}

\begin{document}
\begin{lem}
Layer 1
\end{lem}

\begin{proof}
Layer 0 stuff
\begin{prop}
Layer 1 starts getting embedded
\begin{lem}
This is an artefact: once you get into the first proof you start nesting no matter what
\end{lem}
\end{prop}

Layer 0 stuff
\begin{lem}
Another layer 1
\end{lem}

Can put layer 0 stuff here 
\begin{proof}
Layer 1 proof
\begin{prop}
Layer 2
\end{prop}
\begin{proof}
We conclude with a layer 2 proof
\end{proof}
\end{proof}

\end{proof}
\end{document}

Explanation: I realized that making every theorem a framedtheorem had bad typesetting side-effects. This new version creates both and redefines the former to the latter within the proof environment, where nesting is both likely to occur. Of course, you can do the same within any other environment.

Answer 1

Here's one approach, that uses etoolbox to tweak the definition of \FrameCommand from within the outermost theorem. A complete MWE follows:

\documentclass{article}
\usepackage[framed]{ntheorem}
\usepackage{framed} % To define \FrameCommand
\usepackage{color} % To define \definecolor
\usepackage{lipsum} % For the dummy text
\definecolor{gray}{rgb}{0.6,0.6,0.6}
\usepackage{etoolbox} % For \appto

\theoremstyle{plain}
\newframedtheorem{thm}{Theorem}[page]

\renewcommand*\FrameCommand{\relax} % By default, no framing occurs
\appto\thm{% ... but within the thm environment, FrameCommand is different
  \renewcommand*\FrameCommand{{\color{gray}\vrule width 5pt \hspace{10pt}}}
}


\begin{document}
\noindent \lipsum[1]
\begin{thm}
  Outer theorem
  \begin{thm}
    Theorem nested once
    \begin{thm}
      Theorem nested twice
    \end{thm}
  \end{thm}
\end{thm}
\end{document}

This results in the following appearance:

You'd need to add a similar call to \appto for each additional environment for which you wanted such nesting behavior. (Unfortunately, the original solution I proposed, using AtBeginEnvironment adds the hook too early in the environment definition, so the FrameCommand gets redefined before it's used in the outermost environment too. This version, using \appto\thm, is slightly less polite in that it hacks the opening of the environment by modifying the \thm command directly, but it adds the hook at the right moment.)