I would like to compare in a figure the steps of a running execution of gradient descent algorithm but taking three possible approaches: batch, mini-batch, and stochastic.

I have found an example of batch gradient descent in Plotting the gradient descent.

\documentclass[tikz, margin=3mm]{standalone}
\usepackage{physics,amsmath} %\vb{}
% \usefonttheme[onlymath]{serif} %\vec{}


every edge/.style = {draw, -{Triangle[angle=60:1pt 3,flex]},
                             bend right=11, blue,ultra thick},
every edge quotes/.style = {font=\scriptsize, inner sep=1pt, 
                            auto, sloped}
\fill (0,0) circle[radius=3pt];
\path[name path=C] foreach \i in {4, 8, 16, 22, 28}
        {(0,0) circle[draw=red!\i, x radius=2*\i mm, y radius=\i mm, rotate=-5]};
\foreach \i in  {4, 8, 16, 22, 28}
    \draw[line width=11.2/\i, draw=white!\i!gray]
        (0,0) circle[x radius=2*\i mm, y radius=\i mm, rotate=-5];
\path[name path=V] (-4,2.4) .. controls + (0,-2) and + (-2,0) .. (0,0);
\draw [name intersections={of=C and V, sort by=C, name=A}]
        (A-5) edge ["${\boldsymbol{\theta}[0]}$"] (A-4)
        (A-4) edge ["${\boldsymbol{\theta}[1]}$"] (A-3)
        (A-3) edge ["${\boldsymbol{\theta}[2]}$"] (A-2);

enter image description here

And I would like to make a comparison like in this figure:

enter image description here

  • Are you able to program the three algorithms? At least in a simple case, say for the function x^2 +4y^2. If "yes", do it; then, either do it with math library in tikz or recuperate the descent steps. Either way, you'll have your drawing. If "no", your drawing will represent nothing.
    – Daniel N
    Feb 8 at 4:40
  • @DanielN, thank you. You are right about programming the method. However, I intend to illustrate with a scheme, like the blue line in the example. I want something like a random walk between two points and somehow controlling the direction because stochastic is more random than mini-batch.
    – pablo
    Feb 8 at 8:22


You must log in to answer this question.

Browse other questions tagged .