You have two options. First one is to \protect
\scalebox
:
\documentclass[a4paper,twoside,12pt]{scrreprt}
\usepackage{graphicx}
%\usepackage{fontspec}
\begin{document}
Test \scalebox{3.0}{$\int$} test.
\begin{table}
\centering
\begin{tabular}{cc}
A & B \\
C & D
\end{tabular}
\caption{Test \protect\scalebox{3.0}{$\int$} test.}
\end{table}
\end{document}
Second one is to use the optional argument of \caption
:
\documentclass[a4paper,twoside,12pt]{scrreprt}
\usepackage{graphicx}
%\usepackage{fontspec}
\begin{document}
Test \scalebox{3.0}{$\int$} test.
\begin{table}
\centering
\begin{tabular}{cc}
A & B \\
C & D
\end{tabular}
\caption[Test $\int$ test.]{Test \scalebox{3.0}{$\int$} test.}
\end{table}
\end{document}

This happens as the contents of \caption
are movable (i.e., fragile) as they may be used in lof
and you have to \protect
commands (from expanding) used inside the argument.
$\displaystyle\int$
in the caption? The enlarged text-style integral one gets via\scalebox{3.0}{$\int$}
is not at all centered on the math axis, making it look accidentally comical. This is an effect you are (I would hope!) not actively trying to create, right? – Mico Aug 3 '14 at 4:31