# Drawing Markov Decision Processes with Tikz

I thought this would be much easier. I am trying to recreate the standard MDP graph that is basically the same as a Markov Chain (I know a lot of posts about that) but with the addition of lines that indicate a non-deterministic action. I know I can set up dummy nodes but I am sure there is a more precise and practical way to do this.

Here is an example:

In red is the line that I can't manage to get. Thanks for your help.

This is my code without the red bit:

\begin{tikzpicture}[auto,node distance=8mm,>=latex,font=\small]

\tikzstyle{round}=[thick,draw=black,circle]

\node[round] (s0) {$s_0$};
\node[round,above right=0mm and 20mm of s0] (s1) {$s_1$};
\node[round,below right=0mm and 20mm of s0] (s2) {$s_2$};

\draw[->] (s0) -- (s1);
\draw[->] (s0) -- (s2);
\draw[->] (s0) [out=40,in=100,loop] to (s0);

\end{tikzpicture}
• Welcome! Can you post the code for the rest of the diagram? Then somebody can help you with the red bit you're stuck on. Much easier than starting from scratch, and much more likely to be actually helpful. – cfr Nov 2 '15 at 14:48
• Maybe tex.stackexchange.com/questions/16210/… or similar would be helpful? You can use the angles library for drawing angles although I don't know if maybe the problem you're having is combining that with the loop? – cfr Nov 2 '15 at 14:58
• Hi thanks for the link. I haven't tried this yet, but I would like to avoid to learn the syntax for another package just for that line if possible, I don't want to waste time on it if not really necessary. – Paolo Izzo Nov 2 '15 at 23:26
• Well, initially you gave no indication what syntax you were currently using. But the code helps. Please make it compilable. You should be able to use the angles pic stuff with any TikZ code. – cfr Nov 3 '15 at 0:19

Time spent on learning something will never go a waste!

As suggested in the comments, you have to use angles and quotes library. Let us say you have 3 points A, B and C. Then the angle ABC can be drawn like this:

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{positioning,angles,quotes}
\begin{document}
\begin{tikzpicture}
\coordinate (B);
\coordinate (A) at (3,3);
\coordinate (C) at (3,0);
\draw (B) -- (C) (B) -- (A);
\path pic[draw, angle radius=6mm,"a",angle eccentricity=1.2] {angle = C--B--A};
\end{tikzpicture}
\end{document}

Here angle radius is the radius of the arc and angle eccentricity is the radial distance of the label from the arc (the value 1 is on the arc)

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{positioning,angles,quotes}
\begin{document}
\begin{tikzpicture}[auto,node distance=8mm,>=latex,font=\small]

\tikzstyle{round}=[thick,draw=black,circle]

\node[round] (s0) {$s_0$};
\node[round,above right=0mm and 20mm of s0] (s1) {$s_1$};
\node[round,below right=0mm and 20mm of s0] (s2) {$s_2$};

\draw[->] (s0) -- (s1);
\draw[->] (s0) -- (s2);
\draw[->] (s0) [out=40,in=100,loop] to coordinate[pos=0.1](aa) (s0);
\path pic[draw, angle radius=6mm,"\vphantom{g}a",angle eccentricity=1.2] {angle = s2--s0--aa};
\end{tikzpicture}
\end{document}

angle radius and angle eccentricity need to be carefully adjusted here since one of the lines is a curve.

Following @user11232 suggestion I spent time to learn some Tikz, and here is a simpler way of doing what you want. Simply add two invisible nodes as the markers for the red bend, and then draw an edge with the bend option. Here is your code with the changes annotated by comments:

\begin{tikzpicture}[auto,node distance=8mm,>=latex,font=\small]
\tikzstyle{round}=[thick,draw=black,circle]

\node[round] (s0) {$s_0$};
\node[round,above right=0mm and 20mm of s0] (s1) {$s_1$};
\node[round,below right=0mm and 20mm of s0] (s2) {$s_2$};

\draw[->] (s0) -- (s1);
% Adding invisible marker node below with "node [very near start, anchor=center] (m1) {}"
\draw[->] (s0) -- node [very near start, anchor=center] (m1) {} (s2);
% Here too, with "node [very near start, anchor=center] (m2) {}"
\draw[->] (s0) [out=40,in=100,loop] to node [very near start, anchor=center] (m2) {} (s0);
% Now, this new path draws directly between our two markers.
% Tweak the angle depending on the situation.
\draw (m1.center) edge [bend right=50] (m2.center);
\end{tikzpicture}

`

For completeness sake, I use the tikz libraries automata and positioning.