Define tcolorbox in math mode

$y_{\mywboxmath{0}}=\mywboxmath{-5x}-5+6$
is display math.

\end{document}


Note that in this case you need that \mywboxmath is in math mode.

You can do this using \tcboxmath from the theorems library of tcolorbox:

\documentclass{article}
\usepackage{tcolorbox}
\tcbuselibrary{theorems}

\newcommand*{\mywbox}{%
\tcboxmath[colback=white, colframe=black, size=fbox, arc=3pt, boxrule=0.8pt]%
}

\begin{document}

This \mywbox{3x} is in line math.
This
$y = \mywbox{-5x} - 5 + 6$
is not in line math.

\end{document}


You may want to abstract things a little bit using a style too, so that you can apply it to other boxes should the need arise:

\documentclass{article}
\usepackage{tcolorbox}
\tcbuselibrary{theorems}

\tcbset{my math box/.style={
colback=white, colframe=black, size=fbox, arc=3pt, boxrule=0.8pt}
}

\newcommand*{\mywbox}{\tcboxmath[my math box]}

\begin{document}

This \mywbox{3x} is in line math.
This
$y = \mywbox{-5x} - 5 + 6$
is not in line math.

\end{document}