Neural Networks with MLPClassifier

Neural networks power face unlock, speech recognition and chatbots. Under the hood, they are built from a very simple unit, the artificial neuron, repeated thousands or billions of times and arranged in layers. Once you understand one neuron and how a network learns, the rest is mostly scale.

In this lesson you will compute a small network by hand with NumPy, see how gradient descent learns, and train scikit-learn's multi-layer perceptron (MLP) on handwritten digits. The next lessons build on this for deep learning.

Neurons and layers

A neuron does three things: it multiplies each input by a weight, adds them up together with a bias, and passes the total through an activation function. If the activation is the sigmoid, a single neuron is exactly logistic regression.

A network connects neurons in layers:

• the input layer is just your features;
• one or more hidden layers of neurons transform them step by step;
• the output layer produces the prediction, for example one probability per class.

Each neuron in one layer feeds every neuron in the next, which is why these are called dense or fully connected layers. Here is a forward pass (input to output) through a tiny network with 3 inputs, 2 hidden neurons and 1 output, written with NumPy.

import numpy as np

def relu(z):
    return np.maximum(0, z)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

x = np.array([0.8, 0.2, 0.5])            # 3 scaled inputs for one student
W1 = np.array([[0.5, -0.6],
               [0.9, 0.4],
               [-0.3, 0.8]])              # weights: 3 inputs -> 2 hidden neurons
b1 = np.array([0.1, -0.2])
W2 = np.array([[1.2], [-0.7]])            # weights: 2 hidden -> 1 output
b2 = np.array([0.05])

hidden = relu(x @ W1 + b1)                # weighted sums, then ReLU
output = sigmoid(hidden @ W2 + b2)        # weighted sum, then sigmoid
print("Hidden layer:", hidden.round(3))
print("Output probability:", output.round(3))

Hidden layer: [0.53 0.  ]
Output probability: [0.665]

x @ W1 is matrix multiplication: it computes all the weighted sums of a layer at once, which is why neural networks run so well on GPUs. The second hidden neuron's sum was negative, so ReLU turned it into 0.

Activation functions

Activations add non-linearity. Without them, a stack of layers would collapse into one big weighted sum, no more powerful than linear regression. Common choices:

How a network learns

At first the weights are random, so predictions are poor. Learning repeats a loop:

1. Forward pass: compute predictions for a batch of training examples.
2. Loss: measure how wrong they are, for example with log loss for classification or squared error for regression.
3. Backpropagation: use the chain rule of calculus to work out, for every weight, how the loss would change if that weight changed slightly. This set of slopes is the gradient.
4. Gradient descent: nudge every weight a small step in the direction that lowers the loss. The step size is the learning rate.

One pass over the whole training set is an epoch. Here is gradient descent learning a single weight w for the rule y = 3x, from noisy data.

import numpy as np

rng = np.random.default_rng(42)
x = rng.uniform(0, 5, 50)
y = 3 * x + rng.normal(0, 0.5, 50)

w = 0.0             # start from a bad guess
learning_rate = 0.02
for step in range(1, 31):
    pred = w * x
    loss = np.mean((pred - y) ** 2)
    gradient = np.mean(2 * (pred - y) * x)   # slope of the loss with respect to w
    w -= learning_rate * gradient            # step downhill
    if step in (1, 2, 5, 10, 30):
        print(f"step {step:>2}: loss {loss:7.3f}, w becomes {w:.3f}")

step  1: loss  80.777, w becomes 1.083
step  2: loss  32.795, w becomes 1.772
step  5: loss   2.314, w becomes 2.667
step 10: loss   0.170, w becomes 2.946
step 30: loss   0.146, w becomes 2.978

The loss falls quickly and w settles near 3. A real network does exactly this for millions of weights at once, with backpropagation computing all the gradients efficiently. Too large a learning rate makes the steps overshoot and the loss jump around; too small makes learning painfully slow. Modern optimisers such as Adam (scikit-learn's default) adapt the step size for each weight automatically.

MLPClassifier on handwritten digits

scikit-learn's MLPClassifier builds and trains a network of dense layers. hidden_layer_sizes=(32,) would mean one hidden layer of 32 neurons; (64, 32), used below, means two hidden layers of 64 and 32 neurons. Networks need scaled inputs, and early_stopping=True holds back 10% of the training data and stops when the score on it stops improving, which saves time and limits overfitting.

import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

model = make_pipeline(StandardScaler(),
                      MLPClassifier(hidden_layer_sizes=(64, 32), early_stopping=True,
                                    max_iter=200, random_state=42))
model.fit(X_train, y_train)
net = model[-1]
print("Weight matrices:", [w.shape for w in net.coefs_])
print("Number of weights and biases:", sum(w.size for w in net.coefs_) + sum(b.size for b in net.intercepts_))
print("Stopped after", net.n_iter_, "epochs")
print(f"Test accuracy: {model.score(X_test, y_test):.3f}")

plt.plot(net.loss_curve_, label="training loss")
plt.plot(net.validation_scores_, label="validation accuracy")
plt.xlabel("Epoch")
plt.legend()
plt.show()

Weight matrices: [(64, 64), (64, 32), (32, 10)]
Number of weights and biases: 6570
Stopped after 53 epochs
Test accuracy: 0.960

The weight matrices are 64 × 64 (pixels to the first hidden layer), 64 × 32 (first to second hidden layer) and 32 × 10 (to the ten digits), plus one bias per neuron. The plot shows the training loss falling and the validation accuracy rising, then levelling off, which is when early stopping ends training.

Useful settings to tune: hidden_layer_sizes, alpha (L2 regularisation, like Ridge), learning_rate_init, batch_size and activation ("relu" by default).

MLPRegressor

MLPRegressor predicts numbers. For small datasets, the "lbfgs" solver often trains faster and more reliably than Adam. Here a network with two hidden layers learns the curved mileage pattern from earlier lessons without being told its shape.

import numpy as np
from sklearn.neural_network import MLPRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
speed = rng.uniform(20, 100, 200).reshape(-1, 1)
mileage = 22 - 0.005 * (speed.ravel() - 55) ** 2 + rng.normal(0, 1, 200)

reg = make_pipeline(StandardScaler(),
                    MLPRegressor(hidden_layer_sizes=(16, 16), solver="lbfgs",
                                 max_iter=2000, random_state=42))
reg.fit(speed, mileage)
test_speeds = np.array([[30], [55], [90]])
print("Predicted:", reg.predict(test_speeds).round(1))
print("True:     ", (22 - 0.005 * (test_speeds.ravel() - 55) ** 2).round(1))

Predicted: [18.6 21.8 15.9]
True:      [18.9 22.  15.9]

When to use scikit-learn's MLP

scikit-learn's networks are great for learning and for small tabular problems. They run on the CPU only and offer just dense layers. For images, audio, text or large datasets you need convolutional, recurrent or transformer layers and GPU training, which is what PyTorch, TensorFlow & Keras provide. And for ordinary tables, gradient boosting is often more accurate and much easier to tune than a neural network.

Common mistakes

• Not scaling the inputs. Unscaled features make training slow and unstable; always put a StandardScaler first.
• Ignoring a ConvergenceWarning. It means training stopped at max_iter before settling. Scale the data, raise max_iter, or use early_stopping=True.
• Making the network huge for a small dataset. More neurons mean more weights to fit and more overfitting; start small and increase only if validation scores improve.
• Forgetting random_state. Weights start random, so results change between runs without it.

Key points

• A neuron computes a weighted sum plus a bias and applies an activation; layers of neurons form a network.
• Non-linear activations such as ReLU let networks learn curved patterns; softmax and sigmoid produce probabilities.
• Networks learn by gradient descent: backpropagation computes how each weight affects the loss, and weights move downhill.
• MLPClassifier and MLPRegressor build dense networks; set hidden_layer_sizes, scale inputs and consider early_stopping=True.
• For images, text and large data, move to PyTorch or Keras; for tables, compare with gradient boosting.

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

for layers in [(8,), (32,), (64, 32)]:
    model = make_pipeline(StandardScaler(),
                          MLPClassifier(hidden_layer_sizes=layers, early_stopping=True,
                                        max_iter=200, random_state=42))
    model.fit(X_train, y_train)
    print(f"{str(layers):<9} epochs {model[-1].n_iter_:>3}   test accuracy {model.score(X_test, y_test):.3f}")

(8,)      epochs 129   test accuracy 0.909
(32,)     epochs  38   test accuracy 0.907
(64, 32)  epochs  53   test accuracy 0.960