Instead of using psfrag
I would simply export the image as pdf+tex
from inkscape.
For example if your image looks like this in inkscape:

you can save it as .pdf
and select the option "omit text in pdf and create latex file" during the export:

This will produce two files: a pdf with just the red ellipse but without the text and a .pdf_tex
file that contains the following code:
%% Creator: Inkscape inkscape 0.92.2, www.inkscape.org
%% PDF/EPS/PS + LaTeX output extension by Johan Engelen, 2010
%% Accompanies image file 'drawing.pdf' (pdf, eps, ps)
%%
%% To include the image in your LaTeX document, write
%% \input{<filename>.pdf_tex}
%% instead of
%% \includegraphics{<filename>.pdf}
%% To scale the image, write
%% \def\svgwidth{<desired width>}
%% \input{<filename>.pdf_tex}
%% instead of
%% \includegraphics[width=<desired width>]{<filename>.pdf}
%%
%% Images with a different path to the parent latex file can
%% be accessed with the `import' package (which may need to be
%% installed) using
%% \usepackage{import}
%% in the preamble, and then including the image with
%% \import{<path to file>}{<filename>.pdf_tex}
%% Alternatively, one can specify
%% \graphicspath{{<path to file>/}}
%%
%% For more information, please see info/svg-inkscape on CTAN:
%% http://tug.ctan.org/tex-archive/info/svg-inkscape
%%
\begingroup%
\makeatletter%
\providecommand\color[2][]{%
\errmessage{(Inkscape) Color is used for the text in Inkscape, but the package 'color.sty' is not loaded}%
\renewcommand\color[2][]{}%
}%
\providecommand\transparent[1]{%
\errmessage{(Inkscape) Transparency is used (non-zero) for the text in Inkscape, but the package 'transparent.sty' is not loaded}%
\renewcommand\transparent[1]{}%
}%
\providecommand\rotatebox[2]{#2}%
\ifx\svgwidth\undefined%
\setlength{\unitlength}{595.27559055bp}%
\ifx\svgscale\undefined%
\relax%
\else%
\setlength{\unitlength}{\unitlength * \real{\svgscale}}%
\fi%
\else%
\setlength{\unitlength}{\svgwidth}%
\fi%
\global\let\svgwidth\undefined%
\global\let\svgscale\undefined%
\makeatother%
\begin{picture}(1,1.41428571)%
\put(0,0){\includegraphics[width=\unitlength,page=1]{drawing.pdf}}%
\put(0.30923193,0.68923333){\color[rgb]{1,1,1}\makebox(0,0)[lb]{\smash{$\mathcal{X}$}}}%
\end{picture}%
\endgroup%
(if you need to adjust the colour or position of the text, you can simply edit this file, just make sure you are not overwriting the adjusted file in case you export again from inkscape)
This file can now easily be included in your latex document
\documentclass{article}
\usepackage{graphicx}
\usepackage{color}
\begin{document}
\begin{figure}[htbp]
\centering
\label{HasaRelation}
\def\svgwidth{.3\textwidth}
\input{drawing.pdf_tex}
\caption{something something}
\end{figure}
\end{document}
After compiling with pdflatex (or xe/lualtex) the result looks like this:

psfrag
if you are doing your figures in inkscape? I would either save them aspdf+latex
or export them astikzpicture
from inkscape.