Detection Models: YOLO-Style & Two-Stage

By the end of this lesson you will understand how modern object detectors are built, from the old sliding-window idea to two-stage models like Faster R-CNN and one-stage YOLO-style models. You will decode a YOLO-style output grid into boxes in NumPy and see how to run a pretrained detector with torchvision.

The first idea: sliding windows

The oldest way to detect objects reuses a classifier. Slide a window across the image at many positions and sizes, and ask the classifier at every position: "is there a face here?". Keep the windows that score high and clean them up with non-maximum suppression.

The simplest "classifier" is template matching: compare each window with a small example image of the object and measure similarity. Here we use normalised cross-correlation (NCC), which scores 1.0 for a perfect match and ignores overall brightness differences.

import numpy as np
from numpy.lib.stride_tricks import sliding_window_view

rng = np.random.default_rng(0)
# a small "logo" template: a dark plus sign on a light square
tpl = np.full((9, 9), 200.0)
tpl[3:6, :] = 40
tpl[:, 3:6] = 40

# a noisy scene with the logo pasted at two places (one of them darker overall)
scene = rng.normal(150, 20, (60, 90))
scene[10:19, 15:24] = tpl + rng.normal(0, 5, tpl.shape)
scene[35:44, 60:69] = tpl * 0.6 + rng.normal(0, 5, tpl.shape)

# normalised cross-correlation at every window position
win = sliding_window_view(scene, tpl.shape)                      # (52, 82, 9, 9)
t = (tpl - tpl.mean()) / tpl.std()
w_mean = win.mean(axis=(2, 3), keepdims=True)
w_std = win.std(axis=(2, 3), keepdims=True)
ncc = ((win - w_mean) / w_std * t).mean(axis=(2, 3))
print("NCC map shape:", ncc.shape, " best score:", round(ncc.max(), 3))

hits = np.argwhere(ncc > 0.8)
for r, c in hits:
    print(f"match at row {r}, col {c}, score {ncc[r, c]:.3f}")

NCC map shape: (52, 82)  best score: 0.998
match at row 10, col 15, score 0.998
match at row 35, col 60, score 0.995

Both copies are found, including the darker one, because NCC subtracts each window's mean and divides by its spread. But sliding windows have big limits: you need many window sizes for objects at different distances, a template cannot handle different poses or kinds of car, and checking thousands of windows with a strong classifier is slow. Classical detectors such as Viola-Jones (faces) and HOG with an SVM (pedestrians) made this approach fast enough for their time.

Two-stage detectors: propose, then classify

The R-CNN family split the job in two:

1. Stage 1, region proposals: quickly suggest a few hundred to a couple of thousand boxes that might contain an object.
2. Stage 2, classification and refinement: for each proposal, a CNN predicts the class and nudges the box to fit better.

Faster R-CNN made both stages part of one network. Its Region Proposal Network (RPN) slides over the CNN's feature map and, at each position, scores a set of preset boxes called anchors (several sizes and shapes, such as tall, square and wide). Anchors that look like objects become proposals.

Two-stage detectors are accurate, especially for small objects, but slower. It is like a hospital's emergency desk: a triage nurse quickly picks out who might need a doctor (stage 1), and doctors examine only those patients carefully (stage 2).

One-stage detectors: YOLO-style grids

YOLO ("You Only Look Once", first published in 2016) asked: why not predict everything in a single pass? The image is divided into an S by S grid. Each grid cell predicts:

• an objectness score: is there an object whose centre falls in this cell?
• the box: the centre as an offset inside the cell (0 to 1), plus width and height;
• class probabilities: which kind of object it is.

The final score of a box is objectness × class probability. Boxes below a threshold are dropped and NMS removes duplicates. Because the whole image goes through the network once, YOLO-style models are fast enough for live video. SSD and RetinaNet are other well-known one-stage designs.

Let us decode a tiny YOLO-style output by hand. Step 1: a fake 4 by 4 grid output for 2 classes (car, bike). Step 2: turn cell offsets into image coordinates. Step 3: keep confident boxes.

import numpy as np

S, img_size = 4, 320                   # 4x4 grid over a 320x320 image, each cell is 80 px
classes = ["car", "bike"]
# per cell: objectness, x offset, y offset, width, height (fraction of image), p(car), p(bike)
out = np.zeros((S, S, 7))
out[1, 2] = [0.92, 0.50, 0.40, 0.30, 0.20, 0.95, 0.05]    # a car centred in cell row 1, col 2
out[3, 0] = [0.81, 0.30, 0.60, 0.10, 0.15, 0.10, 0.90]    # a bike in cell row 3, col 0
out[1, 3] = [0.30, 0.10, 0.50, 0.25, 0.20, 0.70, 0.30]    # a weak, doubtful prediction

cell = img_size / S
for row in range(S):
    for col in range(S):
        obj, dx, dy, w, h, *probs = out[row, col]
        k = int(np.argmax(probs))
        score = obj * probs[k]
        if score < 0.5:                                  # Step 3: confidence threshold
            continue
        cx, cy = (col + dx) * cell, (row + dy) * cell     # Step 2: offsets -> pixels
        bw, bh = w * img_size, h * img_size
        box = np.array([cx - bw / 2, cy - bh / 2, cx + bw / 2, cy + bh / 2]).round(1)
        print(f"{classes[k]:4} score {score:.2f} box xyxy {box}")

car  score 0.87 box xyxy [152.  80. 248. 144.]
bike score 0.73 box xyxy [  8. 264.  40. 312.]

Two confident objects survive; the doubtful prediction (0.30 × 0.70 = 0.21) is dropped. Real YOLO-style models predict at several grid sizes (for example strides of 8, 16 and 32 pixels) so that both small and large objects are covered, and they output thousands of candidates before thresholding and NMS.

Anchors, anchor-free models and transformers

Early YOLO versions and Faster R-CNN predicted boxes as adjustments to anchors: preset box shapes chosen to match common object shapes in the training data. Many newer detectors are anchor-free: they predict the box directly from each feature-map position (for example the distances to the four box edges, as in FCOS). This removes anchor tuning.

DETR (DEtection TRansformer) took another route: a transformer outputs a fixed set of boxes, and training matches each prediction to one ground-truth object, so no NMS is needed. Transformer-based detectors are now common in research and are increasingly used in practice.

Running a pretrained detector

torchvision includes detectors pretrained on COCO (a dataset with 80 everyday object categories such as person, car, bus, dog and cup). Run this on your computer; the first run downloads the weights.

import torch
from torchvision.io import decode_image
from torchvision.models.detection import fasterrcnn_resnet50_fpn_v2, FasterRCNN_ResNet50_FPN_V2_Weights

weights = FasterRCNN_ResNet50_FPN_V2_Weights.DEFAULT
model = fasterrcnn_resnet50_fpn_v2(weights=weights, box_score_thresh=0.6)
model.eval()

img = decode_image("street.jpg")                     # uint8 tensor (3, H, W)
batch = [weights.transforms()(img)]                  # a list of float images, one per photo
with torch.no_grad():
    result = model(batch)[0]                         # a dict for the first image

names = weights.meta["categories"]
for box, label, score in zip(result["boxes"], result["labels"], result["scores"]):
    print(f"{names[label]:12} {score:.2f} {box.round().tolist()}")

Each result is a dictionary with boxes (xyxy pixels), labels (category numbers) and scores. NMS has already been applied inside the model. For live video, a lighter model such as torchvision's SSDlite or a YOLO-style model is a better fit. The popular Ultralytics package wraps YOLO-style models in a short API; note that it is published under the AGPL-3.0 licence, which has strict conditions for commercial use, so read its licence before using it in a product.

Common mistakes

• Using a COCO model for classes it never saw. COCO has no "auto-rickshaw" or "pothole"; fine-tune on your own labels.
• Forgetting model.eval() and torch.no_grad(). Training mode changes outputs and wastes memory.
• Setting the score threshold once and forgetting it. Tune it on validation data for your precision/recall needs.
• Judging speed on a powerful desktop GPU. Measure on the device you will deploy to.

Key points

• Sliding windows with a classifier was the first detection approach; it is simple but slow and rigid.
• Two-stage detectors (Faster R-CNN) propose regions, then classify and refine them.
• One-stage YOLO-style detectors predict objectness, box offsets and class scores for every grid cell in one pass.
• Anchors are preset box shapes; anchor-free and transformer detectors (DETR) avoid them.
• torchvision offers COCO-pretrained detectors that return boxes, labels and scores.

import numpy as np

def box_iou(a, b):
    x1 = np.maximum(a[:, None, 0], b[None, :, 0]); y1 = np.maximum(a[:, None, 1], b[None, :, 1])
    x2 = np.minimum(a[:, None, 2], b[None, :, 2]); y2 = np.minimum(a[:, None, 3], b[None, :, 3])
    inter = np.clip(x2 - x1, 0, None) * np.clip(y2 - y1, 0, None)
    area = lambda b: (b[:, 2] - b[:, 0]) * (b[:, 3] - b[:, 1])
    return inter / (area(a)[:, None] + area(b)[None, :] - inter)

def nms(boxes, scores, thr=0.5):
    order, keep = np.argsort(scores)[::-1], []
    while len(order):
        keep.append(int(order[0]))
        order = order[1:][box_iou(boxes[[order[0]]], boxes[order[1:]])[0] <= thr]
    return keep

S, size = 4, 320
out = np.zeros((S, S, 7))
out[1, 2] = [0.92, 0.50, 0.40, 0.30, 0.20, 0.95, 0.05]
out[3, 0] = [0.81, 0.30, 0.60, 0.10, 0.15, 0.10, 0.90]
out[1, 1] = [0.85, 0.95, 0.42, 0.29, 0.21, 0.93, 0.07]
boxes, scores = [], []
for r in range(S):
    for c in range(S):
        obj, dx, dy, w, h, *p = out[r, c]
        s = obj * max(p)
        if s >= 0.5:
            cx, cy, bw, bh = (c + dx) * size / S, (r + dy) * size / S, w * size, h * size
            boxes.append([cx - bw / 2, cy - bh / 2, cx + bw / 2, cy + bh / 2]); scores.append(s)
for thr in (0.5, 0.3):
    keep = nms(np.array(boxes), np.array(scores), thr)
    print(f"threshold {thr}: {len(boxes)} boxes before NMS, {len(keep)} after")   # 3 then 2