Deep Learning: CNNs, RNNs & Transformers

Deep learning is machine learning with neural networks that have many layers. It is behind photo tagging, voice assistants, live translation and chatbots. The big change compared with classic machine learning is that deep networks learn useful features by themselves from raw pixels, sound or text, instead of relying on features you engineer by hand.

This lesson explains the three architectures you will hear about most: convolutional networks (CNNs) for images, recurrent networks (RNNs) for sequences, and transformers, which now dominate text and much more. Training real deep networks needs the frameworks in the next lesson and usually a GPU, so here you will run the core operation of each architecture in NumPy to see how it works.

Why depth helps

A network with many layers learns a hierarchy of features. In an image network, the first layers detect simple edges and colour blobs, middle layers combine them into textures and shapes (an eye, a wheel), and the last layers recognise whole objects (a face, an autorickshaw). Each layer builds on the one below, which is far more efficient than one huge layer trying to learn everything at once.

Deep learning took off around 2012, when a deep CNN won the ImageNet image recognition challenge by a wide margin. Three things made it possible: large labelled datasets, GPUs (graphics cards) that do the matrix maths in parallel, and better training methods. The price is that deep models need much more data, computing power and tuning than the models earlier in this course. For tables of data, gradient boosting is still usually the better choice.

Convolutional neural networks (CNNs)

A dense layer connects every pixel to every neuron, which ignores the fact that nearby pixels belong together. A convolution instead slides a small grid of weights, called a filter or kernel, across the image. At each position it multiplies the filter with the patch under it and adds up the result. The output, a feature map, is large wherever the patch looks like the filter's pattern. The same small filter is reused everywhere, so a CNN needs far fewer weights and finds a pattern wherever it appears.

Here a hand-made 3 × 3 filter that responds to vertical edges is slid over an 8 × 8 picture of a handwritten digit.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
image = digits.images[1]              # 8x8 pixels, values 0 to 16
print("Digit:", digits.target[1])
print(image.astype(int))

kernel = np.array([[-1, 0, 1],
                   [-1, 0, 1],
                   [-1, 0, 1]])        # dark-to-bright from left to right

def convolve(img, k):
    rows, cols = img.shape[0] - 2, img.shape[1] - 2
    out = np.zeros((rows, cols))
    for i in range(rows):
        for j in range(cols):
            out[i, j] = np.sum(img[i:i + 3, j:j + 3] * k)   # patch times filter
    return out

feature_map = convolve(image, kernel)
print("Feature map:")
print(feature_map.astype(int))

pooled = np.maximum(feature_map, 0).reshape(3, 2, 3, 2).max(axis=(1, 3))
print("After ReLU and 2x2 max pooling:")
print(pooled.astype(int))

Digit: 1
[[ 0  0  0 12 13  5  0  0]
 [ 0  0  0 11 16  9  0  0]
 [ 0  0  3 15 16  6  0  0]
 [ 0  7 15 16 16  2  0  0]
 [ 0  0  1 16 16  3  0  0]
 [ 0  0  1 16 16  6  0  0]
 [ 0  0  1 16 16  6  0  0]
 [ 0  0  0 11 16 10  0  0]]
Feature map:
[[  3  38  42 -18 -45 -20]
 [ 18  35  30 -25 -48 -17]
 [ 19  40  29 -36 -48 -11]
 [ 17  41  31 -37 -48 -11]
 [  3  48  45 -33 -48 -15]
 [  2  43  46 -21 -48 -22]]
After ReLU and 2x2 max pooling:
[[38 42  0]
 [41 31  0]
 [48 46  0]]

Large positive values mark places where brightness rises from left to right (the left edge of the stroke) and large negative values where it falls (the right edge). Pooling then shrinks the map by keeping only the strongest response in each 2 × 2 block, which makes the network smaller and less sensitive to small shifts.

In a real CNN, nobody designs the filters: the network learns dozens of them per layer by gradient descent, exactly like the weights in Neural Networks with MLPClassifier. A typical image classifier stacks convolution, ReLU and pooling layers several times, then ends with dense layers and a softmax. Strictly, deep learning libraries compute "cross-correlation" (the filter is not flipped), but everyone calls it convolution.

You can see the effect of different filters with SciPy's ready-made function:

import matplotlib.pyplot as plt
import numpy as np
from scipy.signal import correlate2d
from sklearn.datasets import load_digits

image = load_digits().images[0]          # a handwritten 0
vertical = np.array([[-1, 0, 1]] * 3)
horizontal = vertical.T

fig, axes = plt.subplots(1, 3, figsize=(9, 3))
for ax, (title, picture) in zip(axes, [("Digit", image),
                                       ("Vertical edges", correlate2d(image, vertical, mode="valid")),
                                       ("Horizontal edges", correlate2d(image, horizontal, mode="valid"))]):
    ax.imshow(picture, cmap="gray")
    ax.set_title(title)
    ax.axis("off")
plt.show()

Recurrent neural networks (RNNs)

Text, speech, sensor readings and sales figures are sequences, where order matters. An RNN reads a sequence one step at a time and keeps a hidden state, a small vector that acts as its memory. At each step it combines the new input with the previous hidden state using the same weights, so it can handle sequences of any length.

import numpy as np

rng = np.random.default_rng(42)
W_x = rng.normal(0, 0.5, (1, 3))    # input -> hidden (1 input, 3 memory units)
W_h = rng.normal(0, 0.5, (3, 3))    # previous hidden -> hidden (the "memory" weights)
b = np.zeros(3)

temperatures = [31, 33, 35, 34, 30]   # daily maximum in Jaipur, in °C
h = np.zeros(3)                       # empty memory before the first day
for day, t in enumerate(temperatures, start=1):
    x = np.array([(t - 32) / 3])      # scale the input
    h = np.tanh(x @ W_x + h @ W_h + b)
    print(f"day {day}: {t}°C -> hidden state {h.round(3)}")

day 1: 31°C -> hidden state [-0.051  0.172 -0.124]
day 2: 33°C -> hidden state [ 0.091 -0.203  0.108]
day 3: 35°C -> hidden state [ 0.135 -0.485  0.345]
day 4: 34°C -> hidden state [-0.013 -0.245  0.292]
day 5: 30°C -> hidden state [-0.243  0.483 -0.126]

These weights are random, so the numbers mean nothing yet; the point is the mechanism. Each hidden state depends on today's input and on everything before it. Training adjusts W_x and W_h so that the final state holds what is needed, for example to predict tomorrow's temperature.

Plain RNNs struggle to remember things from many steps back, because the learning signal shrinks as it flows backwards through many steps (the vanishing gradient problem). LSTM (long short-term memory, 1997) and GRU (gated recurrent unit, 2014) cells add learned "gates" that decide what to keep, forget and output, and for years they powered speech recognition and translation. For most text tasks they have now been replaced by transformers, though they are still used for some time-series and on-device problems.

Transformers and attention

The transformer, introduced in the 2017 paper "Attention Is All You Need", drops recurrence entirely. Text is first split into tokens (words or word pieces), and each token becomes a vector called an embedding. Then self-attention lets every token look at every other token and take a weighted mix of them. In "Priya deposited cash in the bank", attention lets "bank" draw information from "deposited" and "cash", so the model understands a money bank, not a river bank.

Each token produces a query ("what am I looking for?"), a key ("what do I contain?") and a value ("what do I pass on?"). The attention weights are the softmax of query-key similarities, and the output is the weighted sum of the values. Here is that calculation on made-up 4-number embeddings.

import numpy as np

def softmax(z):
    e = np.exp(z - z.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

tokens = ["Priya", "deposited", "cash", "bank"]
E = np.array([[1.0, 0.0, 0.2, 0.0],      # toy embeddings, invented for illustration
              [0.1, 1.0, 0.8, 0.6],
              [0.0, 0.7, 1.0, 0.9],
              [0.0, 0.6, 0.9, 1.0]])
Q, K, V = E, E, E    # real models multiply E by learned matrices to get Q, K and V

weights = softmax(Q @ K.T / np.sqrt(E.shape[1]))
print("Attention weights (each row adds up to 1):")
for token, row in zip(tokens, weights):
    print(f"{token:<10}", row.round(2))
print("New vector for 'bank':", (weights @ V)[3].round(2))

Attention weights (each row adds up to 1):
Priya      [0.34 0.23 0.22 0.22]
deposited  [0.12 0.3  0.3  0.28]
cash       [0.11 0.28 0.31 0.3 ]
bank       [0.11 0.27 0.31 0.31]
New vector for 'bank': [0.14 0.67 0.83 0.75]

The row for "bank" puts most of its weight on itself, "cash" and "deposited", and the least on "Priya", so its new vector is mostly a blend of the money-related words. With trained embeddings and learned projections, these differences become much sharper. A real transformer runs many such attention heads in parallel, adds information about word order (positional encoding), and stacks dozens of these layers. Because all tokens are processed at once rather than one by one, transformers train efficiently on GPUs, which is what made today's very large models possible.

Transformers come in two main shapes. Encoders (like BERT) read a whole text and are good at understanding tasks such as classification and search. Decoders (like GPT) predict the next token and are good at generating text; they are the basis of LLMs & Generative AI. The same design now works for images (vision transformers), speech and more.

Transfer learning

Training a large network from scratch needs huge datasets and costly hardware. Instead, people usually start from a pretrained model, one already trained on millions of images or billions of words, and fine-tune it on their own smaller dataset. A CNN pretrained on everyday photos can be fine-tuned to spot crop diseases from a few thousand leaf photos. This transfer learning is the normal way to use deep learning in practice.

Common mistakes

• Using deep learning for small tables. With a few thousand rows of tabular data, gradient boosting is usually more accurate, faster and easier to explain.
• Training from scratch when a pretrained model exists. Fine-tuning is cheaper and usually much better.
• Thinking attention means understanding. Attention is a weighted average learned from data; it can still pick up wrong or biased patterns.
• Ignoring the cost. Large models need GPUs, energy and careful monitoring; budget for them.

Key points

• Deep networks learn a hierarchy of features from raw data, but need lots of data and GPUs.
• CNNs slide small learned filters over images, use ReLU and pooling, and reuse the same weights everywhere.
• RNNs read sequences step by step with a hidden state; LSTM and GRU cells handle longer memories.
• Transformers use self-attention so every token can draw on every other token, and they train in parallel.
• In practice, start from a pretrained model and fine-tune it (transfer learning).

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()
image = digits.images[0]
kernel = np.array([[-1, -1, -1],
                   [0, 0, 0],
                   [1, 1, 1]])          # dark-to-bright from top to bottom

def convolve(img, k):
    rows, cols = img.shape[0] - 2, img.shape[1] - 2
    out = np.zeros((rows, cols))
    for i in range(rows):
        for j in range(cols):
            out[i, j] = np.sum(img[i:i + 3, j:j + 3] * k)
    return out

print("Digit:", digits.target[0])
print(image.astype(int))
print(convolve(image, kernel).astype(int))

Digit: 0
[[ 0  0  5 13  9  1  0  0]
 [ 0  0 13 15 10 15  5  0]
 [ 0  3 15  2  0 11  8  0]
 [ 0  4 12  0  0  8  8  0]
 [ 0  5  8  0  0  9  8  0]
 [ 0  4 11  0  1 12  7  0]
 [ 0  2 14  5 10 12  0  0]
 [ 0  0  6 13 10  0  0  0]]
[[ 13   2 -10 -10   9  18]
 [  3 -12 -26 -32 -14  -4]
 [ -5  -7  -9  -4  -2  -2]
 [ -1  -1   0   5   4   3]
 [  3   8  21  18   5  -5]
 [ -9   4  17  10 -10 -19]]