Video Processing Basics

By the end of this lesson you will be able to treat a video as a NumPy array of frames, detect motion by comparing frames, build a simple background model that ignores parked cars, track moving objects from frame to frame, and read and write real video files with OpenCV.

A video is a stack of images

A video is just many images, called frames, shown quickly one after another. The frame rate says how many per second: 25 or 30 fps (frames per second) is common for cameras, 60 fps for smooth sports. So a 10-second clip at 30 fps is 300 images.

In NumPy a colour video is a 4D array with shape (frames, height, width, 3). A grey video is (frames, height, width). Everything you learned about single images works on each frame, and the new axis, time, lets you see motion.

Think of a flip book from school: each page is a still drawing, and flipping fast makes it move. Video processing is reading the flip book one page at a time and noticing what changed.

Step 1: make a 30-frame video of a ball crossing a road. Step 2: check its shape. Step 3: show a few frames side by side.

import numpy as np
import matplotlib.pyplot as plt

T, H, W = 30, 60, 100
video = np.full((T, H, W), 80, dtype=np.uint8)           # Step 1: grey road in every frame
rr, cc = np.ogrid[:H, :W]
for t in range(T):
    cx = 10 + t * 3                                      # the ball moves 3 px right per frame
    video[t][(rr - 30) ** 2 + (cc - cx) ** 2 <= 6 ** 2] = 230

print("video shape (frames, height, width):", video.shape)    # Step 2
print("memory:", video.nbytes, "bytes")

fig, axes = plt.subplots(1, 4, figsize=(12, 2))               # Step 3
for ax, t in zip(axes, (0, 10, 20, 29)):
    ax.imshow(video[t], cmap="gray", vmin=0, vmax=255); ax.set_title(f"frame {t}"); ax.axis("off")
plt.show()

video shape (frames, height, width): (30, 60, 100)
memory: 180000 bytes

Notice the memory: raw video is huge. One minute of full-HD colour video at 30 fps is 1920 × 1080 × 3 × 1,800 bytes, about 11 GB. That is why video files are compressed with codecs such as H.264, and why real-time systems must process each frame quickly. At 30 fps you have about 33 milliseconds per frame.

Motion by frame differencing

The simplest motion detector subtracts the previous frame from the current one. Pixels that did not change give about 0; moving objects give large values. Threshold the difference and you have a motion mask.

import numpy as np

T, H, W = 30, 60, 100
video = np.full((T, H, W), 80, dtype=np.uint8)
rr, cc = np.ogrid[:H, :W]
for t in range(T):
    video[t][(rr - 30) ** 2 + (cc - (10 + t * 3)) ** 2 <= 6 ** 2] = 230
rng = np.random.default_rng(0)
video = np.clip(video + rng.normal(0, 4, video.shape), 0, 255).astype(np.uint8)   # camera noise

for t in (5, 15, 25):
    diff = np.abs(video[t].astype(int) - video[t - 1].astype(int))   # int: avoid uint8 wrap
    motion = diff > 30
    ys, xs = np.nonzero(motion)
    print(f"frame {t}: {motion.sum():3} moving pixels, box x {xs.min()}-{xs.max()}, y {ys.min()}-{ys.max()}")

frame 5:  70 moving pixels, box x 16-31, y 24-36
frame 15:  70 moving pixels, box x 46-61, y 24-36
frame 25:  70 moving pixels, box x 76-91, y 24-36

The motion boxes move right by about 30 pixels every 10 frames, matching the ball's speed of 3 pixels per frame. Two details matter: convert to int before subtracting (remember the uint8 trap), and use a threshold above the noise level. Frame differencing only shows the edges of change: a large, slowly moving object with flat colour shows only its leading and trailing edges.

A background model

A better approach learns what the empty scene looks like, the background, and compares each frame with it. A simple and robust background model is the running average:

background = (1 - alpha) * background + alpha * frame

With a small alpha such as 0.05, the background slowly absorbs things that stay still (a car that parks) but not things that keep moving. It is like a watchman at a housing society gate who stops noticing a car once it has been parked for a while, but still notices every person walking past.

import numpy as np

rng = np.random.default_rng(1)
T, H, W = 120, 60, 100
video = np.full((T, H, W), 70.0)
video[:, 40:55, 5:35] = 150                      # a car parked from the start
for t in range(T):
    x = (t * 2) % W
    video[t, 20:30, x:x + 6] = 220               # a bike crossing again and again
    if t >= 60:
        video[t, 5:15, 70:95] = 200              # a new car parks at frame 60
video += rng.normal(0, 3, video.shape)

alpha = 0.05
background = video[0].copy()
for t in range(1, T):
    if t in (30, 61, 119):
        fg = np.abs(video[t] - background) > 25
        new_car_pixels = fg[5:15, 70:95].sum()
        bike_pixels = fg[20:30].sum()
        print(f"frame {t:3}: foreground {fg.sum():4} px | bike {bike_pixels:3} | new parked car {new_car_pixels:3}")
    background = (1 - alpha) * background + alpha * video[t]

frame  30: foreground  127 px | bike 127 | new parked car   0
frame  61: foreground  323 px | bike  73 | new parked car 250
frame 119: foreground   75 px | bike  75 | new parked car   0

The car that was parked from the start never shows as foreground. The new car is foreground when it arrives at frame 60, and has faded into the background by frame 119 because it stopped moving. The bike stays foreground throughout. OpenCV's createBackgroundSubtractorMOG2 uses a smarter version: a mixture of Gaussian distributions per pixel, which also copes with flickering lights and swaying trees.

Tracking: who is who across frames?

A detector tells you where objects are in each frame. Tracking links them over time, giving each object an ID, so you can count vehicles crossing a line or measure how long a customer waits. The simplest tracker is the centroid tracker: match each new detection to the existing track whose last centre is closest, if it is close enough; otherwise start a new track.

import numpy as np

# centres of detected objects in 5 consecutive frames (from any detector)
frames = [
    [(10, 50), (80, 20)],
    [(14, 51), (77, 24)],
    [(19, 50), (74, 27), (40, 90)],        # a new object appears
    [(23, 49), (44, 88)],                  # object at (74, 27) left the scene
    [(28, 50), (48, 87)],
]

tracks, next_id, max_jump = {}, 1, 15
for f, detections in enumerate(frames):
    unmatched = list(range(len(detections)))
    for tid, (tx, ty) in list(tracks.items()):
        if not unmatched:
            break
        d = [np.hypot(detections[i][0] - tx, detections[i][1] - ty) for i in unmatched]
        j = int(np.argmin(d))
        if d[j] <= max_jump:
            tracks[tid] = detections[unmatched.pop(j)]
    for i in unmatched:                    # anything left over starts a new track
        tracks[next_id] = detections[i]
        next_id += 1
    print(f"frame {f}: " + ", ".join(f"ID {k} at {v}" for k, v in tracks.items()))

frame 0: ID 1 at (10, 50), ID 2 at (80, 20)
frame 1: ID 1 at (14, 51), ID 2 at (77, 24)
frame 2: ID 1 at (19, 50), ID 2 at (74, 27), ID 3 at (40, 90)
frame 3: ID 1 at (23, 49), ID 2 at (74, 27), ID 3 at (44, 88)
frame 4: ID 1 at (28, 50), ID 2 at (74, 27), ID 3 at (48, 87)

Object IDs stay stable as objects move, and the newcomer gets ID 3. This simple version never deletes tracks: ID 2 stays at its last position after it leaves. Real trackers (SORT, DeepSORT, ByteTrack) add motion prediction with a Kalman filter, remove tracks that are not seen for some frames, and use appearance features to avoid swapping IDs when people cross paths.

Reading and writing video with OpenCV

cv2.VideoCapture reads frames from a file or a camera (cv2.VideoCapture(0) opens the first webcam). cv2.VideoWriter saves frames. This script writes a synthetic clip, then reads it back and finds the ball with MOG2. Run it on your computer.

import cv2
import numpy as np

# Step 1: write a 2-second synthetic video: a white ball moving right on a grey road
fps, w, h = 25, 320, 240
writer = cv2.VideoWriter("ball.mp4", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))  # size is (width, height)
for i in range(2 * fps):
    frame = np.full((h, w, 3), 90, np.uint8)
    cv2.circle(frame, (20 + i * 5, 120), 12, (255, 255, 255), -1)
    writer.write(frame)
writer.release()

# Step 2: read it back frame by frame and find the moving ball with background subtraction
cap = cv2.VideoCapture("ball.mp4")
if not cap.isOpened():
    raise SystemExit("Could not open the video")
print("fps:", cap.get(cv2.CAP_PROP_FPS), " frames:", int(cap.get(cv2.CAP_PROP_FRAME_COUNT)))

subtractor = cv2.createBackgroundSubtractorMOG2(history=100, varThreshold=25, detectShadows=False)
n = 0
while True:
    ok, frame = cap.read()          # ok is False at the end of the video
    if not ok:
        break
    mask = subtractor.apply(frame)
    n += 1
    if n in (10, 40):
        ys, xs = np.nonzero(mask)
        print(f"frame {n}: moving pixels {len(xs)}, centre x about {xs.mean():.0f}")
cap.release()
print("frames read:", n)

fps: 25.0  frames: 50
frame 10: moving pixels 120, centre x about 72
frame 40: moving pixels 455, centre x about 215
frames read: 50

By frame 40 the mask holds about 455 pixels, close to the ball's area (π × 12² ≈ 452), centred where the ball really is (20 + 39 × 5 = 215). At frame 10 the background model is still learning, so its result is rough. Always check ok from cap.read(), and always call release() so the file is closed properly.

Common mistakes

• Subtracting uint8 frames. Convert to int or float first, or differences wrap around.
• Forgetting that VideoWriter wants (width, height). A wrong size silently produces an empty or broken file.
• Not checking ok from cap.read(). At the end of the file, frame is None.
• Running heavy models on every frame. Skip frames, shrink frames, or run detection every few frames and track in between.

Key points

• A video is a stack of frames: (T, H, W) grey or (T, H, W, 3) colour, played at a frame rate such as 25 or 30 fps.
• Frame differencing finds what changed between consecutive frames.
• A running-average background model ignores things that stay still; MOG2 is OpenCV's robust version.
• Tracking links detections over time; a centroid tracker matches each detection to the nearest previous position.
• cv2.VideoCapture reads frames, cv2.VideoWriter writes them; check ok and call release().

import numpy as np

T, H, W = 30, 60, 100
video = np.full((T, H, W), 80, dtype=np.uint8)
rr, cc = np.ogrid[:H, :W]
for t in range(T):
    video[t][(rr - 30) ** 2 + (cc - (10 + t * 3)) ** 2 <= 6 ** 2] = 230

centres = np.array([np.nonzero(frame > 200)[1].mean() for frame in video])
speed = np.diff(centres).mean()
print(f"{speed:.2f} px per frame = {speed * 30:.0f} px per second")   # 2.96 and 89