I have worked it out in Tikz the following way.
First, there are two different inversions:
- the point is inside the inversion circle;
- the point is outside it.
I'm going to start with the second case. I will draw a circle centred on point "O" (called "k") and search the inversion point of point "P" outside the circle "k".
First thing you need to know is one of the tangent point from "P" to "k". The projection of this point to line "OP" is the inversion point of "P".
\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{calc}
\usetikzlibrary{intersections}
\begin{document}
\begin{tikzpicture}
\coordinate (O) at (0,0);
\coordinate (P) at (5,0);
\draw[red,thick,name path=circ1](O)circle(2);
%
% I look for the midpoint of O and P. This point will be the centre of
% an arc whose intersection with k will give me the tangent points
%
\path(O)--coordinate[midway](M)(P);
%
% I draw the arc whose intersection with circle are the 2 tangent points
%
\path[name path=circ2] let
\p1=(O),
\p2=(M),
\n1={veclen(\x2-\x1,\y2-\y1)} in
($(M)+({\n1*cos(130)},{\n1*sin(130)})$) arc (130:230:\n1);
\path[name intersections={of=circ1 and circ2}]
(intersection-1) coordinate (Tg1)
(intersection-2) coordinate (Tg2);
\draw[blue]
(A)--(Tg1)
(O)--node[midway,left,black]{$\mathtt{r}$}(B)
(O)--(A);
%
% Here we are. This projection is the inversion of point P with regards
% to circle k
%
\draw[orange](Tg1)--($(O)!(Tg1)!(P)$)coordinate(InvP);
%
\draw[black,line width=.75,fill=white]
(P)circle(1.5pt)node[black,below]{$\mathtt{P}$};
\draw[black,line width=.75,fill=white]
(O)circle(1.5pt)node[black,below]{$\mathtt{O}$};
\draw[black,line width=.75,fill=white]
(Tg1)circle(1.5pt)node[black,below]{$\mathtt{Tg1}$};
\draw[black,line width=.75,fill=white]
(InvP)circle(1.5pt)node[black,below]{$\mathtt{P'}$};
\node at (-1.75,0) {$\symtt{k}$};
\end{tikzpicture}
\end{document}
When the point is inside the circle, all we have to do is find the perpedicular line from OP, find the intersection of this line with circle k (call it TgP), draw the tangent line from this point and find its intersection with OP.