Gradient Descent

By the end of this lesson you will be able to explain what a gradient is, compute one in two different ways, and write gradient descent, the method that trains every neural network. You will also see what happens when the learning rate is too small or too large.

You met a quick version of this idea in Neural Networks with MLPClassifier. Here we slow down, check every step, and build the tools that backpropagation needs in the next lesson.

Walking downhill in the fog

Imagine you are on a hillside near Munnar in thick fog. You want to reach the valley, but you can only see the ground at your feet. A sensible plan: feel which way the ground slopes, take a small step downhill, and repeat.

That is gradient descent. The "height" is the loss. Your "position" is the current set of weights. The slope tells you which direction makes the loss go up, so you step the other way. The size of each step is the learning rate.

Slopes and derivatives

The derivative of a function tells you its slope at a point: how much the output changes when the input changes a tiny bit. For f(w) = (w - 3)², calculus says the derivative is 2 * (w - 3). At w = 5 the slope is 4 (uphill to the right), so we should move left.

You can also estimate a slope without calculus. Nudge w a tiny bit up and down, and see how much f changes. This is the numerical derivative (or finite difference). It is slow, but it is a perfect way to check your formulas.

Step 1: compare the formula with the numerical slope.

def f(w):
    return (w - 3) ** 2

def slope_formula(w):
    return 2 * (w - 3)

def slope_numerical(func, w, h=1e-5):
    return (func(w + h) - func(w - h)) / (2 * h)   # rise over run

for w in [0.0, 2.0, 3.0, 5.0]:
    print(f"w = {w}: formula {slope_formula(w):6.2f}, numerical {slope_numerical(f, w):6.2f}")

w = 0.0: formula  -6.00, numerical  -6.00
w = 2.0: formula  -2.00, numerical  -2.00
w = 3.0: formula   0.00, numerical   0.00
w = 5.0: formula   4.00, numerical   4.00

The two columns agree. A negative slope means "go right to go down", a positive slope means "go left", and a slope of 0 means you are at the bottom.

Gradient descent on one weight

The update rule is one line:

w = w - learning_rate * slope

Subtracting the slope moves you downhill. Near the bottom the slope gets small, so the steps get small too, and you settle gently.

Priya runs a vegetable stall in Rohtak. She records the weight of each order and what the customer paid. We want the network to learn the price per kilogram, w, so that price = w * kg. The loss is the mean squared error. Its derivative with respect to w is mean(2 * (w*kg - price) * kg).

Step 2: learn the price per kilogram.

import numpy as np

kg = np.array([0.5, 1.0, 1.5, 2.0, 3.0])
price = np.array([21.0, 39.0, 61.0, 80.0, 118.0])   # rupees, true rate is about 40/kg

w = 0.0
learning_rate = 0.05
for step in range(1, 21):
    pred = w * kg
    loss = np.mean((pred - price) ** 2)
    grad = np.mean(2 * (pred - price) * kg)
    w = w - learning_rate * grad
    if step in (1, 2, 3, 5, 10, 20):
        print(f"step {step:>2}: loss {loss:9.2f}, slope {grad:8.2f}, new w {w:6.2f}")

step  1: loss   5201.40, slope  -262.00, new w  13.10
step  2: loss   2335.51, slope  -175.54, new w  21.88
step  3: loss   1049.02, slope  -117.61, new w  27.76
step  5: loss    212.26, slope   -52.80, new w  34.34
step 10: loss      4.95, slope    -7.13, new w  38.97
step 20: loss      1.10, slope    -0.13, new w  39.68

Within a few steps w climbs from 0 to about 39.7 rupees per kg, and the loss falls from over 5,000 to about 1. It cannot reach 0, because real prices are a little noisy. Each step was smaller than the one before, because the slope shrank as w approached the best value.

Choosing the learning rate

The learning rate is the most important setting in training. Think of it as your stride on the foggy hill:

• Too small: you shuffle forward and need forever.
• About right: you reach the valley quickly.
• Too large: you leap over the valley to the other side, higher than before, and the loss explodes.

Step 3: try three learning rates and plot the loss.

import numpy as np
import matplotlib.pyplot as plt

kg = np.array([0.5, 1.0, 1.5, 2.0, 3.0])
price = np.array([21.0, 39.0, 61.0, 80.0, 118.0])

def train(learning_rate, steps=15):
    w, history = 0.0, []
    for _ in range(steps):
        pred = w * kg
        history.append(np.mean((pred - price) ** 2))
        w -= learning_rate * np.mean(2 * (pred - price) * kg)
    return w, history

for lr in [0.005, 0.05, 0.35]:
    w, history = train(lr)
    print(f"lr {lr:<5}: final w = {w:12.2f}, final loss = {history[-1]:14.2f}")
    plt.plot(history, label=f"lr = {lr}")

plt.yscale("log")
plt.xlabel("step")
plt.ylabel("loss (log scale)")
plt.legend()
plt.show()

lr 0.005: final w =        15.70, final loss =        2033.31
lr 0.05 : final w =        39.60, final loss =           1.17
lr 0.35 : final w =      2319.13, final loss =     9991368.51

With 0.005 the weight is still far from 40 after 15 steps. With 0.05 it has arrived. With 0.35 every step overshoots further, and the weight swings to huge positive and negative values. If your training loss ever shoots up to enormous numbers or nan, a learning rate that is too high is the first suspect.

Many weights: the gradient

Real networks have thousands or millions of weights. The gradient is simply the list of slopes, one for each weight. Each slope is a partial derivative: how the loss changes when you nudge that one weight and keep the others fixed. Gradient descent updates every weight at once, each by its own slope.

Let us learn two parameters, a rate w and a fixed delivery charge b, for Priya's home-delivery orders.

Step 4: two parameters, with a gradient check.

import numpy as np

kg = np.array([0.5, 1.0, 1.5, 2.0, 3.0])
price = np.array([40.0, 60.0, 79.0, 101.0, 139.0])   # about 40/kg plus 20 delivery

def loss_fn(w, b):
    return np.mean((w * kg + b - price) ** 2)

def gradients(w, b):
    error = w * kg + b - price
    return np.mean(2 * error * kg), np.mean(2 * error)   # d/dw, d/db

# gradient check at one point: formulas versus nudging
h = 1e-5
dw, db = gradients(10.0, 5.0)
dw_num = (loss_fn(10.0 + h, 5.0) - loss_fn(10.0 - h, 5.0)) / (2 * h)
db_num = (loss_fn(10.0, 5.0 + h) - loss_fn(10.0, 5.0 - h)) / (2 * h)
print(f"check d/dw: {dw:.4f} vs {dw_num:.4f}")
print(f"check d/db: {db:.4f} vs {db_num:.4f}")

w, b, lr = 0.0, 0.0, 0.05
for step in range(1, 2001):
    dw, db = gradients(w, b)
    w, b = w - lr * dw, b - lr * db
    if step in (1, 10, 100, 2000):
        print(f"step {step:>4}: w = {w:6.2f}, b = {b:6.2f}, loss = {loss_fn(w, b):8.3f}")

check d/dw: -245.0000 vs -245.0000
check d/db: -125.6000 vs -125.6000
step    1: w =  16.35, b =   8.38, loss = 2832.492
step   10: w =  39.49, b =  20.22, loss =    0.720
step  100: w =  39.74, b =  20.21, loss =    0.516
step 2000: w =  39.76, b =  20.19, loss =    0.516

The gradient check passes, so the formulas are right. Training then finds a rate near 40 per kg and a delivery charge near 20. Both parameters moved together, each by its own slope. In big networks, different weights often need very different step sizes, which is exactly what the Adam optimiser handles for you.

Valleys, saddles and flat ground

Our loss surfaces were simple bowls with one bottom. A neural network's loss surface is a landscape with millions of dimensions. It can have:

• Local minima: small dips that are not the lowest point.
• Saddle points: places that slope down one way and up another, like a horse saddle, where the gradient is near zero.
• Plateaus: flat regions where progress is slow.

In practice, large networks rarely get stuck in bad local minima. Saddle points and plateaus slow them down more often. The random noise from training on small batches, which you will see in Training Loops, Batches & Epochs, helps the weights wander off these flat spots.

Common mistakes

• Adding the gradient instead of subtracting it. That is gradient ascent, and the loss climbs.
• Keeping a learning rate that makes the loss blow up. Divide it by 10 and try again.
• Using unscaled inputs. If one feature is in lakhs and another is between 0 and 1, no single learning rate suits both.
• Trusting hand-written gradients without checking them. Compare against the numerical slope.

Key points

• A derivative is a slope; the gradient is the list of slopes for every weight.
• Gradient descent repeats w = w - learning_rate * gradient to walk downhill on the loss.
• A numerical gradient check, (f(w+h) - f(w-h)) / 2h, verifies hand-written gradients.
• Too small a learning rate is slow; too large makes the loss explode.
• Deep networks face saddle points and plateaus; batch noise and good optimisers help.

def f(w):
    return w ** 4 - 3 * w ** 2 + w

def slope(w):
    return 4 * w ** 3 - 6 * w + 1

for start in [2.0, -2.0]:
    w = start
    for _ in range(500):
        w -= 0.01 * slope(w)
    print(f"start {start:>4}: w = {w:.3f}, f(w) = {f(w):.3f}")

start  2.0: w = 1.131, f(w) = -1.070
start -2.0: w = -1.301, f(w) = -3.514