Loss Functions

By the end of this lesson you will be able to measure how wrong a network is with a single number, called the loss. You will compute the three losses used in most projects, mean squared error and two kinds of cross-entropy, and match them with PyTorch's built-in versions.

A network cannot improve until it knows how badly it is doing. The loss is that score. Training, in the next lessons, is nothing more than changing the weights to make this number smaller.

What a loss function is

A loss function compares the network's predictions with the true answers and returns one number. Zero means perfect. Bigger means worse. We usually average the loss over a batch of examples.

Think of a cricket coach at a nets session in Jaipur. After each over, the coach gives the bowler one penalty score: a wide is a small penalty, a full toss hit for six is a big one. The bowler adjusts to lower the score. The network does the same with its loss.

Different problems need different losses:

Mean squared error for numbers

Mean squared error (MSE) takes each error (prediction minus truth), squares it, and averages. Squaring makes every error positive and punishes big errors much more than small ones.

Kabir's network predicts second-hand phone prices in thousands of rupees.

Step 1: compute MSE and compare it with MAE.

import numpy as np

true_price = np.array([12.0, 8.5, 20.0, 15.0])     # in thousands of rupees
predicted  = np.array([11.0, 9.0, 26.0, 15.5])

errors = predicted - true_price
mse = np.mean(errors ** 2)
mae = np.mean(np.abs(errors))                        # mean absolute error, for comparison
print("errors:", errors)
print("squared errors:", errors ** 2)
print(f"MSE = {mse:.3f}")
print(f"MAE = {mae:.3f}")

errors: [-1.   0.5  6.   0.5]
squared errors: [ 1.    0.25 36.    0.25]
MSE = 9.375
MAE = 2.000

One bad prediction (₹26,000 for a ₹20,000 phone) gives a squared error of 36, which dominates the MSE. Mean absolute error (MAE) treats all errors in proportion. MSE is the usual default because its slope is smooth and easy to work with, but MAE is calmer when your data has odd outliers.

Binary cross-entropy for yes or no

For a yes/no question the network outputs a probability p that the answer is "yes". Binary cross-entropy (also called log loss) is:

loss = -(y * log(p) + (1 - y) * log(1 - p))

Read it in two halves. If the true answer y is 1, the loss is -log(p): small when p is near 1, huge when p is near 0. If y is 0, the loss is -log(1 - p). The key idea is that confident wrong answers are punished hardest.

Imagine a spam filter at a Pune startup. An email is truly spam (y = 1). Here is the loss for different predictions.

Step 2: see how confidence changes the loss.

import numpy as np

y = 1                                   # the email really is spam
for p in [0.99, 0.9, 0.7, 0.5, 0.3, 0.1, 0.01]:
    loss = -(y * np.log(p) + (1 - y) * np.log(1 - p))
    print(f"predicted p(spam) = {p:<5} loss = {loss:.3f}")

def bce(y_true, p, eps=1e-12):
    p = np.clip(p, eps, 1 - eps)        # avoid log(0)
    return -np.mean(y_true * np.log(p) + (1 - y_true) * np.log(1 - p))

print("batch BCE:", round(bce(np.array([1, 0, 1, 0]), np.array([0.9, 0.2, 0.6, 0.4])), 4))

predicted p(spam) = 0.99  loss = 0.010
predicted p(spam) = 0.9   loss = 0.105
predicted p(spam) = 0.7   loss = 0.357
predicted p(spam) = 0.5   loss = 0.693
predicted p(spam) = 0.3   loss = 1.204
predicted p(spam) = 0.1   loss = 2.303
predicted p(spam) = 0.01  loss = 4.605
batch BCE: 0.3375

Saying 0.5 ("no idea") costs about 0.69. Saying 0.01 (very sure it is not spam) costs 4.6. This strong push against confident mistakes is what makes cross-entropy train classifiers so well. The np.clip line stops log(0), which would give infinity.

Cross-entropy for many classes

With several classes, the network gives one probability per class through softmax. Cross-entropy then looks only at the probability given to the correct class and takes -log of it. If the correct class got probability 0.8, the loss is -log(0.8) = 0.223.

Labels are usually stored as class numbers, such as 2 for the third class. NumPy's fancy indexing, probs[np.arange(n), labels], picks the right probability from each row.

Step 3: cross-entropy on a batch of three digit images.

import numpy as np

def softmax(z):
    e = np.exp(z - z.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)

logits = np.array([[2.0, 0.5, 0.1],      # raw scores for classes 0, 1, 2
                   [0.2, 3.0, 0.3],
                   [1.5, 1.4, 0.2]])
labels = np.array([0, 1, 2])             # the true class of each example

probs = softmax(logits)
picked = probs[np.arange(len(labels)), labels]
print("probabilities:\n", probs.round(3))
print("probability of the true class:", picked.round(3))
print("loss per example:", (-np.log(picked)).round(3))
print("mean cross-entropy:", round(-np.log(picked).mean(), 4))

probabilities:
 [[0.728 0.163 0.109]
 [0.054 0.887 0.06 ]
 [0.459 0.416 0.125]]
probability of the true class: [0.728 0.887 0.125]
loss per example: [0.317 0.12  2.078]
mean cross-entropy: 0.8385

The first two examples are confidently right and get small losses. The third is wrong: the true class 2 got only about 0.12, so its loss is above 2. The average is the batch loss.

Logits and a stable log-softmax

The raw scores before softmax are called logits. Computing softmax and then log separately can lose precision when a probability is tiny. A safer way computes the log of softmax directly with the log-sum-exp trick: log_softmax(z) = z - max(z) - log(sum(exp(z - max(z)))).

Step 4: cross-entropy straight from logits.

import numpy as np

def log_softmax(z):
    shifted = z - z.max(axis=1, keepdims=True)
    return shifted - np.log(np.exp(shifted).sum(axis=1, keepdims=True))

def cross_entropy(logits, labels):
    log_p = log_softmax(logits)
    return -log_p[np.arange(len(labels)), labels].mean()

logits = np.array([[2.0, 0.5, 0.1],
                   [0.2, 3.0, 0.3],
                   [1.5, 1.4, 0.2]])
labels = np.array([0, 1, 2])
print("cross-entropy from logits:", round(cross_entropy(logits, labels), 4))

huge = np.array([[1000.0, 0.0, -1000.0]])
print("very confident and right:", cross_entropy(huge, np.array([0])) + 0.0)   # + 0.0 tidies -0.0
print("very confident and wrong:", cross_entropy(huge, np.array([2])))

cross-entropy from logits: 0.8385
very confident and right: 0.0
very confident and wrong: 2000.0

It gives the same answer as Step 3, and it stays correct even for extreme logits: a loss of 0 when very sure and right, and a huge 2,000 when very sure and wrong. This is exactly why PyTorch's classification losses take logits, not probabilities.

Loss versus metric

You may ask: why not just train on accuracy? Accuracy only changes when a prediction flips from wrong to right, so tiny weight changes usually do not move it at all. It has no useful slope. The loss changes smoothly with every weight, so it can guide learning. We train on the loss and report metrics like accuracy to humans.

The same losses in PyTorch

PyTorch has all three. Note two important details: nn.BCEWithLogitsLoss and nn.CrossEntropyLoss take raw logits and apply sigmoid or softmax inside, and nn.CrossEntropyLoss takes class numbers as labels. Plain code:

import torch
from torch import nn

mse = nn.MSELoss()
print("MSE:", mse(torch.tensor([11.0, 9.0, 26.0, 15.5]),
                  torch.tensor([12.0, 8.5, 20.0, 15.0])).item())

ce = nn.CrossEntropyLoss()
logits = torch.tensor([[2.0, 0.5, 0.1], [0.2, 3.0, 0.3], [1.5, 1.4, 0.2]])
labels = torch.tensor([0, 1, 2])
print("cross-entropy:", round(ce(logits, labels).item(), 4))

bce = nn.BCEWithLogitsLoss()
print("BCE from logits:", round(bce(torch.tensor([2.0, -1.0]), torch.tensor([1.0, 0.0])).item(), 4))

MSE: 9.375
cross-entropy: 0.8385
BCE from logits: 0.2201

The numbers match our NumPy versions. Your hand-written code and the library agree.

Common mistakes

• Putting softmax before nn.CrossEntropyLoss. The loss applies softmax again, and training becomes slow and strange. Pass raw logits.
• Using MSE for classification. It works poorly with sigmoid and softmax outputs. Use cross-entropy.
• Labels in the wrong form. nn.CrossEntropyLoss wants class numbers of type long, such as tensor([0, 2, 1]). nn.BCEWithLogitsLoss wants float labels of the same shape as the logits.
• Taking log(0). In your own NumPy code, clip probabilities or work from logits.

Key points

• The loss is one number that says how wrong the predictions are; training makes it smaller.
• Use MSE for numbers, binary cross-entropy for yes/no and cross-entropy for many classes.
• Cross-entropy punishes confident wrong answers hardest.
• Compute classification losses from logits with log-softmax for numerical safety.
• PyTorch's CrossEntropyLoss and BCEWithLogitsLoss expect raw logits, not probabilities.

import numpy as np

def bce(y_true, p):
    p = np.clip(p, 1e-12, 1 - 1e-12)
    return -np.mean(y_true * np.log(p) + (1 - y_true) * np.log(1 - p))

rained = np.array([1, 0, 1, 1, 0])
model = np.array([0.8, 0.3, 0.6, 0.9, 0.2])
lazy = np.full(5, 0.6)
print(f"model BCE: {bce(rained, model):.4f}")
print(f"lazy  BCE: {bce(rained, lazy):.4f}")

model BCE: 0.2838
lazy  BCE: 0.6730