Classifying Images with Classical ML

By the end of this lesson you will be able to train a real image classifier in your browser, turn images into feature vectors, build a simple version of HOG (histogram of oriented gradients) features, and explain why moving a digit by just one pixel can confuse a model. This is the bridge from hand-written rules to learning from data.

From rules to learning

So far you have written every rule yourself: "threshold at Otsu's value", "a shape with extent 0.78 is a circle". That works for clean shapes. But try writing rules that tell a handwritten 4 from a 9. Every person writes differently, and the rules explode.

Machine learning flips the approach. You collect many labelled examples ("this image is a 4") and let an algorithm find the rules. If you have done the Machine Learning tutorial, the workflow is the same; only the input is an image.

Think of how a child learns to read number plates in Delhi traffic. Nobody gives the child a rulebook. They see hundreds of examples, get corrected, and learn the patterns.

The digits dataset

scikit-learn ships a small dataset of handwritten digits: 1,797 grey images, each 8 by 8 pixels, with values from 0 to 16. It loads instantly in the browser.

Step 1: load it. Step 2: check the shapes. Step 3: look at some examples.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits

# Step 1
digits = load_digits()
# Step 2
print("images:", digits.images.shape, " labels:", digits.target.shape)
print("pixel range:", digits.images.min(), "to", digits.images.max())
print("first image, label", digits.target[0])
print(digits.images[0].astype(int))

# Step 3
fig, axes = plt.subplots(1, 8, figsize=(10, 1.8))
for ax, img, label in zip(axes, digits.images, digits.target):
    ax.imshow(img, cmap="gray_r"); ax.set_title(label); ax.axis("off")
plt.show()

images: (1797, 8, 8)  labels: (1797,)
pixel range: 0.0 to 16.0
first image, label 0
[[ 0  0  5 13  9  1  0  0]
 [ 0  0 13 15 10 15  5  0]
 [ 0  3 15  2  0 11  8  0]
 [ 0  4 12  0  0  8  8  0]
 [ 0  5  8  0  0  9  8  0]
 [ 0  4 11  0  1 12  7  0]
 [ 0  2 14  5 10 12  0  0]
 [ 0  0  6 13 10  0  0  0]]

You can almost read the 0 in the printed numbers: high values form a ring around a low-value centre. cmap="gray_r" draws high values dark, like ink on paper.

Pixels as features

Most classical models expect each example to be a flat row of numbers, called a feature vector. The simplest feature vector for an image is all its pixels in one row: an 8 by 8 image becomes 64 numbers. We train a logistic regression model on 75% of the images and test it on the other 25%, which it has never seen.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split

digits = load_digits()
X_tr, X_te, y_tr, y_te = train_test_split(digits.images, digits.target,
                                          test_size=0.25, random_state=0, stratify=digits.target)
flat = lambda imgs: imgs.reshape(len(imgs), -1) / 16      # 8x8 -> 64 numbers in 0..1

model = LogisticRegression(max_iter=2000).fit(flat(X_tr), y_tr)
print("test accuracy:", round(model.score(flat(X_te), y_te), 3))
print(confusion_matrix(y_te, model.predict(flat(X_te))))

test accuracy: 0.969
[[45  0  0  0  0  0  0  0  0  0]
 [ 0 45  0  0  0  0  0  0  1  0]
 [ 0  1 43  0  0  0  0  0  0  0]
 [ 0  0  0 45  0  0  0  1  0  0]
 [ 0  0  0  0 43  0  0  0  2  0]
 [ 0  1  0  0  0 45  0  0  0  0]
 [ 0  2  0  0  0  0 43  0  0  0]
 [ 0  0  0  0  0  0  0 45  0  0]
 [ 0  3  0  0  0  1  0  1 38  0]
 [ 0  0  0  0  0  1  0  0  0 44]]

About 97% of unseen digits are classified correctly. The confusion matrix shows the details: row = true digit, column = predicted digit. Numbers off the diagonal are mistakes; the most common one here is an 8 predicted as a 1. You will study these tools in Evaluating Vision Models.

The weakness: shift the image by one pixel

Pixel features have a hidden problem. The model learned "pixel 27 is usually dark for a 0". If the digit moves one pixel to the right, every pixel value lands in a different position, and the learned weights no longer line up. Let us test it.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

digits = load_digits()
X_tr, X_te, y_tr, y_te = train_test_split(digits.images, digits.target,
                                          test_size=0.25, random_state=0, stratify=digits.target)
flat = lambda imgs: imgs.reshape(len(imgs), -1) / 16

def shift(imgs, dx):
    """Move every image dx pixels right (negative = left), filling with 0."""
    out = np.zeros_like(imgs)
    if dx > 0:
        out[:, :, dx:] = imgs[:, :, :-dx]
    else:
        out[:, :, :dx] = imgs[:, :, -dx:]
    return out

model = LogisticRegression(max_iter=2000).fit(flat(X_tr), y_tr)
for dx in (0, 1, 2):
    test = X_te if dx == 0 else shift(X_te, dx)
    print(f"shifted {dx} px right: accuracy {model.score(flat(test), y_te):.3f}")

shifted 0 px right: accuracy 0.969
shifted 1 px right: accuracy 0.396
shifted 2 px right: accuracy 0.056

A one-pixel shift drops accuracy from about 97% to about 40%, and two pixels makes the model nearly useless. A person would not even notice the difference. This is the central problem of image classification: the same object can appear at different positions, sizes, angles and lighting, and raw pixels change completely.

Better features: a simple HOG

Before deep learning, the fix was to design features that change less when the image moves a little. A famous one is HOG, the histogram of oriented gradients (Dalal and Triggs, 2005), which was the standard for pedestrian detection.

The idea uses tools you already know:

1. Compute gradients gx, gy (as in Edge Detection), and their magnitude and angle.
2. Split the image into small cells (here 4 by 4 pixels, so 4 cells).
3. In each cell, make a histogram of gradient angles (8 bins), where each pixel votes with its gradient magnitude.
4. Normalise each histogram and join them into one feature vector.

The histogram says "this cell has strong vertical strokes" without caring exactly which pixel they are in. That makes small shifts matter less.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

def shift(imgs, dx):
    out = np.zeros_like(imgs)
    if dx > 0:
        out[:, :, dx:] = imgs[:, :, :-dx]
    else:
        out[:, :, :dx] = imgs[:, :, -dx:]
    return out

def hog_like(imgs, cell=4, bins=8):
    feats = []
    for im in imgs:
        gx = np.zeros_like(im); gy = np.zeros_like(im)
        gx[:, 1:-1] = im[:, 2:] - im[:, :-2]                 # Step 1: gradients
        gy[1:-1, :] = im[2:, :] - im[:-2, :]
        mag = np.hypot(gx, gy)
        ang = np.degrees(np.arctan2(gy, gx)) % 180           # direction, ignoring sign
        b = np.minimum((ang / (180 / bins)).astype(int), bins - 1)
        f = []
        for r in range(0, 8, cell):                          # Step 2: cells
            for c in range(0, 8, cell):
                h = np.bincount(b[r:r + cell, c:c + cell].ravel(),        # Step 3: weighted histogram
                                weights=mag[r:r + cell, c:c + cell].ravel(), minlength=bins)
                f.append(h / (np.linalg.norm(h) + 1e-6))      # Step 4: normalise
        feats.append(np.concatenate(f))
    return np.array(feats)

digits = load_digits()
X_tr, X_te, y_tr, y_te = train_test_split(digits.images, digits.target,
                                          test_size=0.25, random_state=0, stratify=digits.target)
F_tr = hog_like(X_tr)
print("HOG feature vector length:", F_tr.shape[1])
model = LogisticRegression(max_iter=2000).fit(F_tr, y_tr)
for dx in (0, 1, 2):
    test = X_te if dx == 0 else shift(X_te, dx)
    print(f"shifted {dx} px right: accuracy {model.score(hog_like(test), y_te):.3f}")

HOG feature vector length: 32
shifted 0 px right: accuracy 0.884
shifted 1 px right: accuracy 0.624
shifted 2 px right: accuracy 0.244

The results are a trade-off. On unshifted digits our tiny HOG scores lower than raw pixels, because 8 by 8 images are too small for good gradient histograms. But after shifts it holds up much better: about 62% instead of 40% at one pixel. Real HOG uses bigger images, overlapping blocks and careful normalisation.

Why this leads to CNNs

Designing features by hand is slow and never perfect. Each new problem (faces, leaves, X-rays) needed new expert features. Convolutional neural networks solve both problems at once:

• Their first layers are convolution kernels, like the Sobel filters you coded, but learned from data.
• Pooling layers summarise small regions, which gives tolerance to small shifts, much like HOG's cells.
• Deeper layers combine simple features into complex ones: edges into corners, corners into shapes, shapes into objects.

There is also a simpler fix you can apply even to raw pixels: show the model shifted copies during training. That is data augmentation, and it has its own lesson.

Common mistakes

• Testing on training images. Always keep a test set the model never saw.
• Forgetting to scale pixels. Divide by the maximum (16 here, 255 for photos) so features are on a similar scale.
• Trusting accuracy on perfectly aligned data. Real photos are shifted, rotated and lit differently; test for that.
• Spending weeks hand-designing features. For anything beyond simple, controlled images, try a CNN with transfer learning first.

Key points

• Machine learning learns rules from labelled examples instead of hand-written rules.
• Flattening pixels gives a feature vector; a linear model already reaches about 97% on aligned digits.
• Pixel features break when the object moves even one pixel.
• HOG features (histograms of gradient directions in cells) are more tolerant of small shifts.
• CNNs learn their own convolution features and use pooling for shift tolerance.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

def shift(imgs, dx):
    out = np.zeros_like(imgs)
    if dx > 0:
        out[:, :, dx:] = imgs[:, :, :-dx]
    else:
        out[:, :, :dx] = imgs[:, :, -dx:]
    return out

digits = load_digits()
X_tr, X_te, y_tr, y_te = train_test_split(digits.images, digits.target,
                                          test_size=0.25, random_state=0, stratify=digits.target)
flat = lambda imgs: imgs.reshape(len(imgs), -1) / 16
X_aug = np.concatenate([X_tr, shift(X_tr, 1), shift(X_tr, -1)])
y_aug = np.tile(y_tr, 3)
model = LogisticRegression(max_iter=3000).fit(flat(X_aug), y_aug)
print("unshifted:", round(model.score(flat(X_te), y_te), 3))
print("shifted 1:", round(model.score(flat(shift(X_te, 1)), y_te), 3))