CNN Architectures: LeNet to ResNet

By the end of this lesson you will be able to read and build the classic CNN designs. You will build a LeNet-style network, understand why VGG uses many small 3 × 3 kernels, and write a residual block, the idea that made networks with over a hundred layers trainable.

Each famous architecture solved one specific problem. Learn the problem, and the design makes sense. You already know the building blocks from Convolution & Pooling; here we stack them.

The family tree

Think of how Indian Railways coaches changed over the decades. Each new design kept what worked (wheels, bogies, couplings) and fixed one clear problem (speed, safety, comfort). CNN history is the same: the basic parts stay, the arrangement improves.

Step 1: a LeNet-style network

LeNet's pattern is the template for every CNN: feature extractor (convolution and pooling blocks) followed by a classifier (dense layers). Here is a modern version for 28 × 28 grey images, with ReLU and max pooling where the original used older choices. This is plain code with its real output.

import torch
from torch import nn

class LeNet(nn.Module):
    def __init__(self, n_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 6, kernel_size=5, padding=2), nn.ReLU(), nn.MaxPool2d(2),   # 28 -> 14
            nn.Conv2d(6, 16, kernel_size=5), nn.ReLU(), nn.MaxPool2d(2),            # 14 -> 10 -> 5
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(16 * 5 * 5, 120), nn.ReLU(),
            nn.Linear(120, 84), nn.ReLU(),
            nn.Linear(84, n_classes),
        )

    def forward(self, x):
        return self.classifier(self.features(x))

model = LeNet()
x = torch.randn(2, 1, 28, 28)
print("features:", tuple(model.features(x).shape))
print("logits:  ", tuple(model(x).shape))
print("parameters:", sum(p.numel() for p in model.parameters()))

features: (2, 16, 5, 5)
logits:   (2, 10)
parameters: 61706

The feature extractor turns each 28 × 28 image into 16 small 5 × 5 maps. The classifier flattens those 400 numbers and decides. About 60,000 parameters: tiny by today's standards, and still a solid digit reader.

AlexNet: depth, ReLU and GPUs

AlexNet followed the same pattern but much bigger: five convolutional layers, three dense layers and around 60 million parameters, trained on 1.2 million ImageNet photos. Its lessons were practical ones that you have already used in this course:

• ReLU instead of sigmoid or tanh, so gradients did not vanish (see Activation Functions).
• Dropout in the dense layers against overfitting.
• Data augmentation: random crops and flips of the training photos.
• GPUs: training was split across two graphics cards.

VGG: small kernels, stacked

VGG asked a simple question: what if every convolution is 3 × 3? Two 3 × 3 layers in a row "see" a 5 × 5 patch of the input, and three see 7 × 7. The area of the input that affects one output value is called the receptive field.

Stacking small kernels gives the same receptive field as one big kernel, with fewer weights and more non-linearity (a ReLU after each layer).

Step 2: compare one big kernel with stacked small ones.

def conv_params(channels, k):
    return channels * channels * k * k        # weights only, same channels in and out

def receptive_field(n_layers, k=3):
    return 1 + n_layers * (k - 1)             # stride 1, no pooling

channels = 64
for big_k, n_small in [(5, 2), (7, 3)]:
    one_big = conv_params(channels, big_k)
    stacked = n_small * conv_params(channels, 3)
    print(f"{big_k}x{big_k} receptive field: one {big_k}x{big_k} layer = {one_big:,} weights, "
          f"{n_small} stacked 3x3 = {stacked:,} weights "
          f"(field {receptive_field(n_small)}x{receptive_field(n_small)})")

5x5 receptive field: one 5x5 layer = 102,400 weights, 2 stacked 3x3 = 73,728 weights (field 5x5)
7x7 receptive field: one 7x7 layer = 200,704 weights, 3 stacked 3x3 = 110,592 weights (field 7x7)

Three stacked 3 × 3 layers need about 45% fewer weights than one 7 × 7 layer, for the same view of the input. That is why almost every CNN since has used 3 × 3 kernels as its default.

The problem with going deeper

If depth helps, why not 100 layers? Researchers tried, and found something surprising: beyond about 20 layers, plain CNNs got worse, even on their training data. This was not overfitting. The deeper networks were simply harder to optimise. The gradient had to pass back through so many layers that it became tiny or unstable.

Here is a toy view. Suppose each layer multiplies the backward signal by its local slope. In a plain chain, the slopes multiply together. In a residual chain, each layer computes x + f(x), so its slope is 1 + f'(x): the "1" from the skip lets the signal through even when f'(x) is small.

Step 3: follow the gradient through 50 layers.

import numpy as np

rng = np.random.default_rng(0)
layer_slopes = rng.uniform(0.3, 0.9, size=50)       # small local slopes, as in a deep net

plain = np.cumprod(layer_slopes)                     # gradient reaching each depth
residual = np.cumprod(1 + 0.02 * (layer_slopes - 0.6))   # 1 + a small change per layer

for depth in (1, 10, 25, 50):
    print(f"after {depth:>2} layers: plain {plain[depth - 1]:.2e}   residual {residual[depth - 1]:.3f}")

after  1 layers: plain 6.82e-01   residual 1.002
after 10 layers: plain 5.59e-03   residual 1.006
after 25 layers: plain 5.42e-07   residual 0.998
after 50 layers: plain 3.00e-12   residual 1.016

Through 50 plain layers, the signal shrinks to almost nothing, so the early layers barely learn. Through 50 residual layers, it stays close to 1. Real networks are more complicated, but this is the core reason skip connections work.

ResNet: the residual block

ResNet (He and colleagues, 2015) builds the network from residual blocks. Each block computes some layers F(x) and then adds the original input back: output = relu(F(x) + x). The connection that carries x around the layers is a skip connection (or shortcut).

Think of editing an essay. Instead of rewriting the whole essay at each pass, each editor only writes corrections on top of the previous version. If an editor has nothing useful to add, the essay passes through unchanged. A residual block learns a correction to its input, and "no change" is easy to learn.

When the block changes the number of channels or halves the size, x no longer has the same shape as F(x). Then the shortcut uses a 1 × 1 convolution with the same stride to reshape x, so the two can be added.

import torch
from torch import nn

class ResidualBlock(nn.Module):
    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.body = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, 3, stride=stride, padding=1, bias=False),
            nn.BatchNorm2d(out_ch), nn.ReLU(),
            nn.Conv2d(out_ch, out_ch, 3, padding=1, bias=False),
            nn.BatchNorm2d(out_ch),
        )
        self.shortcut = nn.Identity()                          # pass x through unchanged
        if stride != 1 or in_ch != out_ch:                     # reshape x when needed
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_ch, out_ch, 1, stride=stride, bias=False), nn.BatchNorm2d(out_ch))

    def forward(self, x):
        return torch.relu(self.body(x) + self.shortcut(x))

x = torch.randn(2, 32, 16, 16)
print("same shape:   ", tuple(ResidualBlock(32, 32)(x).shape))
print("down-sampling:", tuple(ResidualBlock(32, 64, stride=2)(x).shape))

same shape:    (2, 32, 16, 16)
down-sampling: (2, 64, 8, 8)

Stacking such blocks, with the channels doubling and the size halving every few blocks, gives the ResNet family: ResNet-18, 34, 50, 101 and 152, named after their number of layers. torchvision includes them, and you can inspect one without downloading any weights:

import torch
from torchvision.models import resnet18

model = resnet18(weights=None)            # the architecture only, random weights
print("parameters:", f"{sum(p.numel() for p in model.parameters()):,}")
print("last layers:", model.avgpool, "|", model.fc)
print("output:", tuple(model(torch.randn(1, 3, 224, 224)).shape))

parameters: 11,689,512
last layers: AdaptiveAvgPool2d(output_size=(1, 1)) | Linear(in_features=512, out_features=1000, bias=True)
output: (1, 1000)

Notice AdaptiveAvgPool2d: global average pooling reduces each final feature map to one number, so the classifier is a single small Linear layer from 512 features to the 1,000 ImageNet classes.

After ResNet

Later designs refined these ideas: EfficientNet scaled depth, width and image size together; MobileNet made CNNs light enough for phones; ConvNeXt modernised ResNet with ideas from transformers. Vision transformers split an image into patches and use the attention you will meet in Attention & Transformers. In practice you rarely design an architecture from scratch. You pick a proven one, often pretrained, as the next lesson on Transfer Learning shows.

Common mistakes

• Adding a skip connection with mismatched shapes. If channels or size change, the shortcut needs a 1 × 1 convolution with the matching stride.
• Using bias=True in a convolution followed by batch norm. The bias is wasted, since batch norm subtracts the mean.
• Designing a new architecture for an ordinary task. Start with a ResNet or another proven model.
• Hard-coding the flatten size. Print the feature shape once, or use global average pooling so any image size works.

Key points

• Classic CNNs have a feature extractor (convolution and pooling) followed by a classifier.
• AlexNet showed that deep CNNs with ReLU, dropout, augmentation and GPUs beat older methods on ImageNet.
• VGG's stacked 3 × 3 kernels give a large receptive field with fewer weights and more non-linearity.
• Very deep plain networks are hard to optimise; ResNet's skip connections, relu(F(x) + x), keep gradients flowing.
• Modern CNNs end with global average pooling and a small linear layer.

def receptive_field(n_layers, k=3):
    return 1 + n_layers * (k - 1)

n = 1
while receptive_field(n) < 15:
    n += 1
stacked = n * 32 * 32 * 3 * 3
single = 32 * 32 * 15 * 15
print(f"{n} layers give a {receptive_field(n)}x{receptive_field(n)} field")
print(f"stacked 3x3: {stacked:,} weights, one 15x15: {single:,} weights")

7 layers give a 15x15 field
stacked 3x3: 64,512 weights, one 15x15: 230,400 weights