Beamer frame option to change background and font that works with labels

Question

I'd like to define a frame option in Beamer to "invert" the colors of background, foreground, and structure, and I can do so by defining

\documentclass{beamer}

\BeforeBeginEnvironment{frame}{%
  \setbeamercolor{normal text}{fg=black, bg=white}%
  \setbeamercolor{structure}{fg=blue}%
  \usebeamercolor[fg]{normal text}%
}

\makeatletter
\define@key{beamerframe}{bg}[true]{%
  \setbeamercolor{normal text}{fg=white, bg=black}%
  \setbeamercolor{structure}{fg=red}%
  \usebeamercolor[fg]{normal text}%
}
\makeatother

\begin{document}
  \begin{frame}
    \frametitle{Normal}
  \end{frame}

  \begin{frame}[bg]
    \frametitle{Inverted}
    A statement
  \end{frame}
\end{document}

which results in

However, assigning the frame a label

\begin{frame}[bg, label=test]
  \frametitle{Inverted}
  A statement
\end{frame}

results in the error

Package keyval Error: label undefined \end{frame}

Now this is already a problem, but we could say only a minor problem, since it can be circumvented by placing the label first.

Things really start to spiral out of my control when I'd like to reuse the frame

\begin{document}
  \begin{frame}<1>[label=test, bg]
    \frametitle{Inverted}
    \only<1>{A statement}
    \only<2>{Another statement}
    \only<3>{Third statement}
  \end{frame}

  \againframe<2>{test}

  \begin{frame}
    \frametitle{Normal}
  \end{frame}

  \againframe<3>{test}
\end{document}

Somehow the frame option affects all contiguous frames, but not if there's a "normal" frame in-between.

The behaviour I was expecting to see, is that the inversion would not affect the repeated frames, but instead the frame option would need to be passed again. However, doing that

\againframe<3>[bg]{test}

results in the error

Package keyval Error: relabel undefined 

If changing only the frame background following the example of https://tex.stackexchange.com/a/395429/199342, the label order issue disappears, as does the issue of not being able to pass the option to \againframe. Nevertheless, the frame option still affects contiguous frames even without passing the option, but passing it does not result in any errors either.

\documentclass{beamer}

\defbeamertemplate*{background canvas}{mydefault}{%
  \ifbeamercolorempty[bg]{background canvas}{}{\color{bg}\vrule width\paperwidth height\paperheight}%
}

\defbeamertemplate*{background canvas}{bg}{%
  \color{black}\vrule width\paperwidth height\paperheight%
}

\BeforeBeginEnvironment{frame}{%
  \setbeamertemplate{background canvas}[mydefault]%
}
\makeatletter
\define@key{beamerframe}{bg}[true]{%
  \setbeamertemplate{background canvas}[bg]%
}
\makeatother

\begin{document}
  \begin{frame}<1>[bg, label=test]
    \frametitle{Inverted}
    \only<1>{First point}
    \only<2>{Second point}
    \only<3>{Third point}
  \end{frame}

  \againframe<2>{test}

  \begin{frame}
    \frametitle{Normal}
  \end{frame}

  \againframe<3>[bg]{test}
\end{document}

But I'd really like the frame option to change the font colors as well. Is there some way to achieve the functionality I'm looking for?