# Batch vs mini batch vs stochastic gradient descent

I would like to compare in a figure the steps of a running execution of gradient descent algorithm but taking three possible approaches: batch, mini-batch, and stochastic.

I have found an example of batch gradient descent in Plotting the gradient descent.

\documentclass[tikz, margin=3mm]{standalone}
\usepackage{physics,amsmath} %\vb{}
%
% \usefonttheme[onlymath]{serif} %\vec{}

\usetikzlibrary{arrows.meta,
bending,
intersections,
quotes,
shapes.geometric}

\begin{document}
\begin{tikzpicture}[
every edge/.style = {draw, -{Triangle[angle=60:1pt 3,flex]},
bend right=11, blue,ultra thick},
every edge quotes/.style = {font=\scriptsize, inner sep=1pt,
auto, sloped}
]
\path[name path=C] foreach \i in {4, 8, 16, 22, 28}
\foreach \i in  {4, 8, 16, 22, 28}
\draw[line width=11.2/\i, draw=white!\i!gray]
\path[name path=V] (-4,2.4) .. controls + (0,-2) and + (-2,0) .. (0,0);
%
\draw [name intersections={of=C and V, sort by=C, name=A}]
(A-5) edge ["${\boldsymbol{\theta}[0]}$"] (A-4)
(A-4) edge ["${\boldsymbol{\theta}[1]}$"] (A-3)
(A-3) edge ["${\boldsymbol{\theta}[2]}$"] (A-2);
\end{tikzpicture}
\end{document}


And I would like to make a comparison like in this figure:

• Are you able to program the three algorithms? At least in a simple case, say for the function x^2 +4y^2. If "yes", do it; then, either do it with math library in tikz or recuperate the descent steps. Either way, you'll have your drawing. If "no", your drawing will represent nothing. Commented Feb 8, 2023 at 4:40
• @DanielN, thank you. You are right about programming the method. However, I intend to illustrate with a scheme, like the blue line in the example. I want something like a random walk between two points and somehow controlling the direction because stochastic is more random than mini-batch. Commented Feb 8, 2023 at 8:22