Very trivial with a token cycle, which treats each token of the argument successively. All characters in the input are processed, one at a time, through the \Characterdirective
. I assume that content is characters alone...all macros, spaces, and groups will be discarded from the input. If punctuation finds its way into the input, it will just be echoed.
\documentclass{article}
\usepackage{amssymb,tokcycle}
\Characterdirective{%
\tctestifcatnx A#1{\ifnum`#1<`[ \mathcal{#1}\else\mathfrak{#1}\fi}{#1}%
}
\newcommand\mixedfont[1]{\tokcyclexpress{#1}}
\begin{document}
$y = \mixedfont{AbcDef}$
\end{document}

If you want the token cycle to handle macros and groups, you just need to tell it what to do when it encounters them in the input. And when you deal with macros that take arguments, you can no longer output the transformed tokens directly (as I did before), but instead gather them in a token list, and output the token list at the end. If you try to output them directly, then the macros will be expecting their arguments before you process them.
\documentclass{article}
\usepackage{amssymb,tokcycle}
\newcommand\mixedfont[1]{\tokcycle
{\tctestifcatnx A##1{\ifnum`##1<`[ \addcytoks{\mathcal{##1}}\else
\addcytoks{\mathfrak{##1}}\fi}{\addcytoks{##1}}}% CHARACTERS
{\processtoks{##1}}% GROUPS
{\addcytoks{##1}}% MACROS
{\addcytoks{##1}}% SPACES
{#1}\the\cytoks}
\begin{document}
$y = \mixedfont{A\frac{b}{c}D^{2x+1}ef}$
\end{document}
