Gradient descent

Roll downhill on a loss curve. Find the learning rate that converges — and the one that explodes.

Current w
Earlier steps
Tangent (slope = f′(w))

Tap or drag on the chart to choose the starting point.

Training a model means finding the weight w that makes the loss small. Gradient descent looks at the slope of the loss where it stands (the red tangent) and takes a step downhill, scaled by the learning rate η.

Too small an η and progress crawls; too big and each step jumps over the valley, landing higher than before until it explodes. On a bumpy curve, where you start decides which valley you end up in.

w ← w − η · f′(w) f(w) = (w − 3)² + 1

Loss curve

Learning rate η0.1

Starting point w₀-1.0

Step: 0
w: -1.000
Loss f(w): 17.000
Gradient f′(w): -8.000
Step size η·f′(w): -0.800
Status: Ready

Too small an η and progress crawls; too big and each step jumps over the valley, landing higher than before until it explodes. On a bumpy curve, where you start decides which valley you end up in.

w ← w − η · f′(w) f(w) = (w − 3)² + 1

Things to try

On the simple bowl, find the learning rate that lands on the minimum in a single step.
Set η = 0.95 on the bowl. Why does w zig-zag from side to side? What happens at η = 1.05?
On the bumpy curve, start at w = 3. Do you reach the deepest valley? Change only the start to get there.
With a tiny η = 0.01, how many steps does it take to get close? Why is this a problem for big models?

Test yourself: 5-minute drill Next: Overfitting & model complexity