Layers & the Forward Pass

By the end of this lesson you will be able to build a layer of many neurons with one matrix, push a batch of examples through several layers, and predict the shape of every result. You will also count how many weights a network has.

The forward pass means running the input through the network, layer by layer, to get a prediction. Almost every error beginners hit in deep learning is a shape error in the forward pass, so this lesson is worth reading slowly.

From one neuron to a layer

A layer is a group of neurons that all read the same inputs. Each neuron has its own weights. Instead of storing them separately, we put them side by side as the columns of one weight matrix W.

Think of a class marks register. Each row is a student and each column is a subject. Now imagine five teachers, each giving every subject a different importance to compute their own "score". Each teacher's importances form one column of W. One matrix multiplication gives every student's score from every teacher at once.

For a layer with n_in inputs and n_out neurons:

A layer where every input connects to every neuron is called a dense or fully connected layer. In PyTorch it is called nn.Linear.

Step 1: one dense layer on a batch. Isha has data for four students: hours studied per day, attendance (0 to 1) and last exam marks (scaled to 0 to 1).

import numpy as np

rng = np.random.default_rng(1)
X = np.array([[2.0, 0.90, 0.72],
              [0.5, 0.60, 0.40],
              [3.5, 0.95, 0.88],
              [1.0, 0.75, 0.55]])        # 4 students, 3 features

W = rng.normal(0, 0.5, size=(3, 5))      # 3 inputs -> 5 neurons
b = np.zeros(5)

Z = X @ W + b
print("X shape:", X.shape)
print("W shape:", W.shape)
print("Z shape:", Z.shape)
print(Z.round(2))

X shape: (4, 3)
W shape: (3, 5)
Z shape: (4, 5)
[[ 0.56  0.78  0.33 -1.2   0.86]
 [ 0.23  0.15  0.11 -0.25  0.22]
 [ 0.83  1.42  0.53 -2.18  1.51]
 [ 0.35  0.36  0.18 -0.56  0.43]]

Z has 4 rows (students) and 5 columns (neurons). Notice that b has shape (5,) but was added to a (4, 5) array. NumPy broadcasting copied the bias row to every student automatically.

Stacking layers

A network is a chain: the output of one layer is the input of the next. Between layers we apply an activation such as ReLU. At the end, softmax turns the last layer's scores into probabilities.

Isha wants three classes: "needs help", "pass" and "distinction". So her network goes 3 → 5 → 4 → 3.

Step 2: a full forward pass with shapes.

import numpy as np

def relu(z):
    return np.maximum(0, z)

def softmax(z):
    e = np.exp(z - z.max(axis=1, keepdims=True))   # one row per example
    return e / e.sum(axis=1, keepdims=True)

rng = np.random.default_rng(1)
X = np.array([[2.0, 0.90, 0.72],
              [0.5, 0.60, 0.40],
              [3.5, 0.95, 0.88],
              [1.0, 0.75, 0.55]])

sizes = [3, 5, 4, 3]
params = [(rng.normal(0, 0.5, size=(n_in, n_out)), np.zeros(n_out))
          for n_in, n_out in zip(sizes[:-1], sizes[1:])]

a = X
for i, (W, b) in enumerate(params, start=1):
    z = a @ W + b
    a = softmax(z) if i == len(params) else relu(z)
    print(f"layer {i}: {W.shape[0]} -> {W.shape[1]}, output shape {a.shape}")

classes = ["needs help", "pass", "distinction"]
print(a.round(3))
print("predicted:", [classes[k] for k in a.argmax(axis=1)])

layer 1: 3 -> 5, output shape (4, 5)
layer 2: 5 -> 4, output shape (4, 4)
layer 3: 4 -> 3, output shape (4, 3)
[[0.315 0.374 0.311]
 [0.331 0.34  0.33 ]
 [0.298 0.411 0.291]
 [0.326 0.351 0.324]]
predicted: ['pass', 'pass', 'pass', 'pass']

Each row of the final array holds three probabilities that add up to 1. The weights are random, so the predictions mean nothing yet. Training, which starts in Gradient Descent, will fix that. Notice axis=1, keepdims=True in softmax: it works across each row (each student), not across the whole batch.

Counting parameters

The weights and biases are the network's parameters: the numbers training will adjust. A dense layer has n_in × n_out weights plus n_out biases.

A classic digit reader takes a 28 × 28 image (784 pixels), has one hidden layer of 128 neurons and outputs 10 classes. How many parameters is that?

Step 3: count parameters.

def count_params(sizes):
    total = 0
    for n_in, n_out in zip(sizes[:-1], sizes[1:]):
        layer = n_in * n_out + n_out
        print(f"{n_in:>4} -> {n_out:<4} {layer:>9,} parameters")
        total += layer
    return total

print("Isha's network:", count_params([3, 5, 4, 3]))
print()
print("Digit reader:", f"{count_params([784, 128, 10]):,}")
print()
print("Wider and deeper:", f"{count_params([784, 512, 512, 10]):,}")

   3 -> 5           20 parameters
   5 -> 4           24 parameters
   4 -> 3           15 parameters
Isha's network: 59

 784 -> 128    100,480 parameters
 128 -> 10       1,290 parameters
Digit reader: 101,770

 784 -> 512    401,920 parameters
 512 -> 512    262,656 parameters
 512 -> 10       5,130 parameters
Wider and deeper: 669,706

The digit reader has about 100,000 parameters. Make the layers 512 wide and you pass 650,000. Dense layers grow fast because every input connects to every neuron. This is one reason images use convolution instead, which shares a few weights across the whole picture.

A reusable layer class

Let us wrap a layer in a small class. Later lessons add a backward pass to it.

Step 4: a Dense class.

import numpy as np

class Dense:
    def __init__(self, n_in, n_out, rng):
        self.W = rng.normal(0, 0.5, size=(n_in, n_out))
        self.b = np.zeros(n_out)

    def forward(self, X):
        return X @ self.W + self.b

rng = np.random.default_rng(1)
layers = [Dense(3, 5, rng), Dense(5, 3, rng)]
X = rng.normal(size=(6, 3))      # 6 examples

a = X
for layer in layers[:-1]:
    a = np.maximum(0, layer.forward(a))
scores = layers[-1].forward(a)
print("scores shape:", scores.shape)

try:
    layers[0].forward(rng.normal(size=(6, 4)))   # wrong number of features
except ValueError as err:
    print("Error:", err)

scores shape: (6, 3)
Error: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 3 is different from 4)

The last lines show the error you will meet often. The input had 4 features, but the layer expects 3, so the "core dimension" of the multiplication does not match. Read such errors as "the inner sizes of @ disagree".

The same thing in PyTorch

PyTorch's nn.Linear(n_in, n_out) is our Dense class. One detail differs: PyTorch stores the weight as (n_out, n_in) and computes X @ W.T + b. You rarely notice, but it explains the shape you see when printing it. This is plain code; run it on your own computer.

import torch
from torch import nn

torch.manual_seed(0)
layer = nn.Linear(3, 5)
X = torch.randn(4, 3)
print("weight shape:", layer.weight.shape)
print("bias shape:  ", layer.bias.shape)
print("output shape:", layer(X).shape)
print("parameters:  ", sum(p.numel() for p in layer.parameters()))

weight shape: torch.Size([5, 3])
bias shape:   torch.Size([5])
output shape: torch.Size([4, 5])
parameters:   20

torch.Size is PyTorch's name for a shape. You will meet nn.Linear properly in Building Models with nn.Module.

Common mistakes

• Swapping the weight shape. In NumPy code, W is (n_in, n_out) for X @ W. Writing (n_out, n_in) breaks the multiplication.
• Feeding one example without a batch dimension. A single example should usually be shape (1, n_in), not (n_in,). Use x.reshape(1, -1).
• Softmax over the wrong axis. On a batch, use axis=1 (per example). axis=0 mixes different examples together.
• Forgetting the activation between layers. The network silently becomes linear, as you saw in Activation Functions.

Key points

• A dense layer is X @ W + b, with W of shape (n_in, n_out) and b of shape (n_out,).
• A batch of shape (batch, n_in) becomes (batch, n_out); the inner sizes must match.
• The forward pass runs layer after layer, with an activation between layers and softmax at the end for classes.
• A dense layer has n_in × n_out + n_out parameters, which grows very fast.
• PyTorch's nn.Linear stores its weight as (n_out, n_in) and computes X @ W.T + b.

import numpy as np

class Dense:
    def __init__(self, n_in, n_out, rng):
        self.W = rng.normal(0, 0.5, size=(n_in, n_out))
        self.b = np.zeros(n_out)

    def forward(self, X):
        return X @ self.W + self.b

rng = np.random.default_rng(3)
sizes = [10, 32, 16, 1]
layers = [Dense(i, o, rng) for i, o in zip(sizes[:-1], sizes[1:])]

a = rng.normal(size=(8, 10))
for layer in layers[:-1]:
    a = np.maximum(0, layer.forward(a))
out = 1 / (1 + np.exp(-layers[-1].forward(a)))
print("output shape:", out.shape)
print("parameters:", sum(l.W.size + l.b.size for l in layers))

output shape: (8, 1)
parameters: 897