First off, I believe that the slashbox
package is deprecated and is no longer part of TeXLive. Instead, use the diagbox
package.
Second, you may fix the problem you've encountered by omitting the @{}
particle in the argument of the tabular
environment. I.e., replace
\begin{tabular}{|@{}c|c|c|c|c|}
with
\begin{tabular}{|c|c|c|c|c|}
More generally, you may want to re-consider your approach to displaying information in a tabular fashion.
When I look at your table, what I see first and foremost are lots and lots of vertical and horizontal lines. It takes a non-trivial effort to start discovering what's inside all those little (prison) cells.
Do think about employing a more open and inviting "look" to present the tabular material. You could begin by omitting all vertical lines and retaining only a few, but well-spaced, horizontal lines, as in done in the example shown below. (By the way, is the third data row a repeat of the first data row? If so, the third row is redundant and could be omitted, right?)
I'm pretty sure your readers will appreciate your effort -- and reward it by actually bothering to look at the contents of the table. Getting your readers to absorb the information you present must be seen as part of having written a successful paper, right? :-)
\documentclass[a4paper,twoside,12pt]{book}
\usepackage{diagbox,amsmath,amssymb}
\usepackage{newtxtext,newtxmath} % mathptmx is very old
\usepackage{booktabs} % new
\usepackage[skip=0.333\baselineskip]{caption} % optional
\begin{document}
\begin{table}[h!]
\centering
\caption{Tabel $Q\cap M'$}
\label{tabelksmanna}
\begin{tabular}{|c|c|c|c|c|}
\hline
\backslashbox{\tabular{@{}l@{}}$Q\in\mathfrak{p}$\endtabular}{$M'\in P(A)$}
& $\varnothing$& $\{1\}$& $\{2\}$&$\{1,2\}$\\
\hline
$\varnothing$&$\varnothing$&$\varnothing$&$\varnothing$&$\varnothing$\\
\hline
$\{1\}$&$\varnothing$&$\{1\}$&$\varnothing$&$\{1\}$\\
\hline\hline
$\varnothing$&$\varnothing$&$\varnothing$&$\varnothing$&$\varnothing$\\
\hline
$\{2\}$&$\varnothing$&$\varnothing$&$\{2\}$&$\{2\}$\\
\hline
\end{tabular}
\bigskip
\caption{Same material, rearranged}
%% Use an 'array' env. since all contents are math-y
$\begin{array}{@{}lcccc@{}}
\toprule
Q\in\mathfrak{p} & \multicolumn{4}{c@{}}{M'\in P(A)}\\
\cmidrule(l){2-5}
& \varnothing & \{1\} & \{2\} &\{1,2\} \\
\midrule
\varnothing&\varnothing&\varnothing&\varnothing&\varnothing\\
\{1\} & \varnothing & \{1\} & \varnothing & \{1\} \\
\addlinespace
\varnothing&\varnothing&\varnothing&\varnothing&\varnothing\\
\{2\}&\varnothing&\varnothing&\{2\}&\{2\}\\
\bottomrule
\end{array}$
\end{table}
\end{document}