In the solution below I've used
\numberwithin{equation}{section} % (1.1), (1.2), etc
\newtheorem{mytheorem}[equation]{Theorem} % Theorems share equation numbers
to get numbering (1.1)
, (1.2)
, etc. and then mytheorem
gets the same numbers as equations
.
I've borrowed some important code from Command behavior depending on current environment so that my newcounter
subequations
can be incremented, but only when within mytheorem
.
I've used the etoolbox
for its important \appto
command.

Complete MWE:
% arara: pdflatex
\documentclass{article}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{etoolbox}
\numberwithin{equation}{section} % (1.1), (1.2), etc
\newtheorem{mytheorem}[equation]{Theorem} % Theorems share equation numbers
% https://tex.stackexchange.com/questions/39738/command-behavior-depending-on-current-environment
\makeatletter
\def\Mycurrentvir{document}
\def\ifenv#1{
\def\@tempa{#1}%
\ifx\@tempa\Mycurrentvir
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
}
\makeatother
\newcounter{subequation}
\appto\mytheorem{\setcounter{subequation}{0}%
\def\Mycurrentvir{mytheorem}%
\renewcommand{\theequation}{\arabic{section}.\arabic{equation}.\arabic{subequation}}}
\AtBeginEnvironment{equation}{\ifenv{mytheorem}{\refstepcounter{subequation}\addtocounter{equation}{-1}}{}}
\begin{document}
\section{My section}
\begin{equation}\label{eq:first}
y=mx+b
\end{equation}
\begin{mytheorem}\label{thm:test}
\begin{equation}\label{eq:test}
y=ax^2+bx+c
\end{equation}
Another equation
\begin{equation}
y=ax^2+bx+c
\end{equation}
\end{mytheorem}
\begin{equation}
y=x^3
\end{equation}
\begin{itemize}
\item First equation: \eqref{eq:first}
\item Equation within theorem: \eqref{eq:test}
\item Theorem reference: \ref{thm:test}
\end{itemize}
\section{another section}
\begin{equation}
y=mx+b
\end{equation}
\begin{mytheorem}
\begin{equation}
y=ax^2+bx+c
\end{equation}
Another equation
\begin{equation}
y=ax^2+bx+c
\end{equation}
\end{mytheorem}
\begin{equation}
y=x^3
\end{equation}
\end{document}