Autograd

By the end of this lesson you will be able to let PyTorch compute every gradient for you with one call, loss.backward(). You will check its answers against the hand-made backpropagation from earlier, understand why gradients add up, and build a tiny autograd engine of your own to see how it works inside.

Autograd is PyTorch's automatic differentiation system. You write only the forward pass. PyTorch quietly records each operation, and when you ask, it walks that record backwards applying the chain rule, exactly as you did in Backpropagation Step by Step.

PyTorch examples are plain code with their real output. The tiny engine in Step 4 is pure Python, so you can run it here.

Step 1: your first gradient

Mark a tensor with requires_grad=True to say "I will want gradients for this". Then compute a result and call .backward() on it. The gradient appears in the tensor's .grad attribute.

We reuse the single neuron from the backpropagation lesson: input x = 1.5, target y = 1, weight w = 0.4, bias b = -0.2, sigmoid activation and squared error.

import torch

x, y = torch.tensor(1.5), torch.tensor(1.0)
w = torch.tensor(0.4, requires_grad=True)
b = torch.tensor(-0.2, requires_grad=True)

a = torch.sigmoid(w * x + b)
loss = (a - y) ** 2
loss.backward()                 # chain rule, all the way back

print(f"loss = {loss.item():.4f}")
print(f"dL/dw = {w.grad.item():.4f}, dL/db = {b.grad.item():.4f}")

loss = 0.1611
dL/dw = -0.2893, dL/db = -0.1928

These are the same numbers you computed by hand: dL/dw = -0.2893 and dL/db = -0.1928. One call replaced four lines of chain rule. For a network with fifty layers, it replaces hundreds.

.item() turns a one-number tensor into an ordinary Python number, which is handy for printing.

The computation graph

While the forward pass runs, PyTorch builds a computation graph: a record of every operation and which tensors it used. Think of a shop's billing ledger in Kochi. Every line says what was added or multiplied and from which earlier lines. To find how one entry affected the final total, you trace the ledger backwards.

Each result tensor remembers the operation that made it in its grad_fn. Tensors you create yourself, like w and b, are leaf tensors: the starting points of the graph. Gradients are stored only on leaves.

Step 2: look at the graph.

import torch

w = torch.tensor(0.4, requires_grad=True)
z = w * 1.5 - 0.2
a = torch.sigmoid(z)
loss = (a - 1.0) ** 2

print("loss made by:", loss.grad_fn.name())
print("a made by:   ", a.grad_fn.name())
print("z made by:   ", z.grad_fn.name())
print("w is a leaf: ", w.is_leaf, " z is a leaf:", z.is_leaf)

loss made by: PowBackward0
a made by:    SigmoidBackward0
z made by:    SubBackward0
w is a leaf:  True  z is a leaf: False

The names read backwards like the ledger: the loss came from a power, which came from a subtraction, which came from a sigmoid, and so on. backward() simply follows these links. The graph is rebuilt on every forward pass, so you can use ordinary Python if statements and loops inside a model.

Step 3: gradients add up

When you call backward(), PyTorch adds the new gradients to whatever is already in .grad. It does not replace them. This is deliberate (it lets you sum gradients from several small batches), but it surprises everyone once.

import torch

w = torch.tensor(2.0, requires_grad=True)
for i in range(3):
    loss = w ** 2                  # slope is 2 * w = 4
    loss.backward()
    print(f"after backward #{i + 1}: w.grad = {w.grad.item()}")

w.grad = None                      # clear it, as optimizer.zero_grad() does
(w ** 2).backward()
print("after clearing:", w.grad.item())

after backward #1: w.grad = 4.0
after backward #2: w.grad = 8.0
after backward #3: w.grad = 12.0
after clearing: 4.0

The true slope is 4, but the stored gradient grew to 8 and 12. In a training loop that means every update uses the sum of all past gradients, and training goes badly wrong. This is why every PyTorch loop calls optimizer.zero_grad() once per batch.

Gradient descent with autograd

Now a complete small training loop using only autograd, no optimiser object yet. This is Priya's vegetable price model from Gradient Descent: learn a rate per kilogram and a delivery charge.

Two new tools appear:

• torch.no_grad(): a block where PyTorch does not record operations. Use it when updating weights and when evaluating, because those steps must not become part of the graph.
• w -= lr * w.grad: an in-place update that keeps w the same leaf tensor.

import torch

kg = torch.tensor([0.5, 1.0, 1.5, 2.0, 3.0])
price = torch.tensor([40.0, 60.0, 79.0, 101.0, 139.0])
w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)

for step in range(1, 2001):
    loss = ((w * kg + b - price) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= 0.05 * w.grad
        b -= 0.05 * b.grad
    w.grad = None
    b.grad = None
    if step in (1, 10, 2000):
        print(f"step {step:>4}: loss {loss.item():9.3f}, w {w.item():6.2f}, b {b.item():6.2f}")

step    1: loss  8192.600, w  16.35, b   8.38
step   10: loss     1.098, w  39.49, b  20.22
step 2000: loss     0.516, w  39.76, b  20.19

The weights match the NumPy version from the gradient descent lesson step for step, but we never wrote a single derivative. (The loss here is measured before each update, so the first value is higher than in that lesson.) In the next lessons, nn.Module will hold the weights and torch.optim will do the update and the clearing.

Step 4: a tiny autograd engine you can run

How does PyTorch do it? Each operation stores its inputs and a small rule for its local slope. backward() visits the operations from the end to the start and multiplies slopes along the way. Here is a toy version for single numbers, written in plain Python. Run it, then compare with Step 1.

import math

class Node:
    def __init__(self, value, parents=(), slopes=()):
        self.value, self.grad = value, 0.0
        self.parents, self.slopes = parents, slopes   # local slope to each parent

    def __add__(self, other):
        other = other if isinstance(other, Node) else Node(other)
        return Node(self.value + other.value, (self, other), (1.0, 1.0))

    def __mul__(self, other):
        other = other if isinstance(other, Node) else Node(other)
        return Node(self.value * other.value, (self, other), (other.value, self.value))

    def __sub__(self, other):
        return self + (other * -1.0 if isinstance(other, Node) else Node(-other))

    def __pow__(self, n):
        return Node(self.value ** n, (self,), (n * self.value ** (n - 1),))

    def sigmoid(self):
        s = 1 / (1 + math.exp(-self.value))
        return Node(s, (self,), (s * (1 - s),))

    def backward(self):
        order, seen = [], set()
        def visit(node):                              # sort: inputs before outputs
            if id(node) not in seen:
                seen.add(id(node))
                for p in node.parents:
                    visit(p)
                order.append(node)
        visit(self)
        self.grad = 1.0
        for node in reversed(order):                  # walk backwards, chain rule
            for parent, slope in zip(node.parents, node.slopes):
                parent.grad += node.grad * slope

w, b = Node(0.4), Node(-0.2)
a = (w * 1.5 + b).sigmoid()
loss = (a - 1.0) ** 2
loss.backward()
print(f"loss = {loss.value:.4f}")
print(f"dL/dw = {w.grad:.4f}, dL/db = {b.grad:.4f}")

loss = 0.1611
dL/dw = -0.2893, dL/db = -0.1928

Forty lines, and it gives the same gradients as PyTorch. Note the += in backward: a node used in two places gets gradient from both, which is the same "gradients add up" behaviour you saw in Step 3. PyTorch does the same thing for whole tensors, on CPUs and GPUs, with hundreds of operation types.

Things that need a scalar, and things that stop gradients

Two more facts you will meet early:

• backward() with no arguments needs a single number, like a mean loss. Calling it on a tensor of many values raises an error. Reduce with .mean() or .sum() first.
• .detach() returns a tensor with the same values but cut off from the graph. Use it to save or plot values, or to stop gradients flowing into part of a model.

import torch

w = torch.tensor([1.0, 2.0], requires_grad=True)
out = w * 3
try:
    out.backward()
except RuntimeError as err:
    print("Error:", err)

out.sum().backward()
print("after sum().backward():", w.grad)
print("detached:", out.detach(), " requires_grad:", out.detach().requires_grad)

Error: grad can be implicitly created only for scalar outputs
after sum().backward(): tensor([3., 3.])
detached: tensor([3., 6.])  requires_grad: False

Common mistakes

• Not clearing gradients. Without optimizer.zero_grad() (or setting .grad = None), gradients from every step pile up.
• Updating weights outside torch.no_grad(). PyTorch either raises an error about a leaf variable or records the update into the graph.
• Calling .numpy() on a tensor that needs gradients. Use t.detach().numpy().
• Evaluating without torch.no_grad(). It works, but wastes memory recording a graph you will never use.
• Calling backward() on a non-scalar. Average or sum the loss first.

Key points

• requires_grad=True asks PyTorch to track a tensor; loss.backward() fills .grad on every leaf.
• PyTorch records a computation graph during the forward pass and walks it backwards with the chain rule.
• Gradients accumulate, so clear them every step with optimizer.zero_grad().
• Use torch.no_grad() for weight updates and evaluation, and .detach() to cut a tensor from the graph.
• Autograd gives exactly the same gradients as careful hand-written backpropagation.

import torch

x = torch.tensor(2.0, requires_grad=True)
(x ** 3 - 4 * x).backward()
print("slope at x = 2:", x.grad.item())       # 3*4 - 4 = 8

x.grad = None
for _ in range(100):
    f = x ** 3 - 4 * x
    f.backward()
    with torch.no_grad():
        x -= 0.05 * x.grad
    x.grad = None
print(f"minimum near x = {x.item():.4f}")

slope at x = 2: 8.0
minimum near x = 1.1547