Gradient Based Optimization Algorithms - Visualized

• These are all the animations used in the Optimization lectures taught by Prof.Mitesh Khapra at IIT Madras in the Deep Learning course.
• I have uploaded the notebook that I used to create these animations (or) you can directly go to colab here
• That notebook has an implementation for the gradient descent algorithm. You need to modify the update rule for each optimization algorithm.
• The objective is to get an intuitive idea of the differences in the optimization algorithms with contrived examples.
• You can find all the animations used in the lecture in .mp4 format in the Animations directory
• Here are a few samples

Gradient Descent

Momentum Gradient Descent

Nesterov Accelarated Gradient Descent

AdaGrad

Adam