how can I create the negation of mathcal letters

Question

I know that the negation of < is got by \not <. The same with many symbols. But this doesn't work with mathcal letters for example, like \not \mathcal{R} or \mathcal{\not R}.

Answer 1

The package cancel can draw a diagonal line.

\documentclass[a4paper]{article}

\usepackage{amsmath}
\usepackage{cancel}
\begin{document}

$\cancel{\mathcal{R}}$

\end{document}

Answer 2

\not is designed for negating relation symbols basically as wide as the equals sign. But it doesn't always work and, for instance, one should use \notin rather than \not\in because the membership sign is too high.

Here's a possible solution for your “negated relations”:

\documentclass{article}
\usepackage{xparse}
\NewDocumentCommand{\calrel}{sm}
 {%
  \mathrel{
   \IfBooleanTF{#1}
   {\notcalrel{#2}}
   {\mathcal{#2}}%
  }%
 }
\NewDocumentCommand{\notcalrel}{m}
 {%
  \ooalign{$\mathcal{#1}$\cr\noalign{\kern-.2ex}\hidewidth$/$\hidewidth\cr}
 }

\begin{document}
We have $x \calrel{R} y$ but $a \calrel*{R} b$.

We have $x \calrel*{R} y$ but $a \calrel{R} b$.

\end{document}

With \calrel{R} you get a relation symbol with the correct spacing around it; the *-variant adds the negation bar.

Note that the *-variant will not work in subscripts or superscripts; this shouldn't be much of a concern, though.

Using this with \mathcal{T} poses some problems: the slope of the glyph is very similar to the slope of the slash and the central bar gets in the way. A solution might be to rotate a bit the slash, making it longer.

\documentclass{article}
\usepackage{xparse,graphicx}
\NewDocumentCommand{\calrel}{sm}
 {%
  \mathrel{
   \IfBooleanTF{#1}
   {\notcalrel{#2}}
   {\mathcal{#2}}%
  }%
 }
\newcommand{\notcalrel}[1]
 {%
  \ooalign{%
   $\mathcal{#1}$\cr
   \noalign{\kern-.05ex}
   \hidewidth\kern.05em\rotatedslash\hidewidth\cr}
 }
\newcommand\rotatedslash{%
  \rotatebox[origin=c]{-20}{\scalebox{1}[1.2]{/}}%
}

\begin{document}
We have $x \calrel{R} y$ but $a \calrel*{R} b$.

We have $x \calrel*{R} y$ but $a \calrel{R} b$.

We have $x \calrel{S} y$ but $a \calrel*{S} b$.

We have $x \calrel*{S} y$ but $a \calrel{S} b$.

We have $x \calrel{T} y$ but $a \calrel*{T} b$.

We have $x \calrel*{T} y$ but $a \calrel{T} b$.

\end{document}