Support Vector Machines

A support vector machine (SVM) looks for the boundary that separates two classes with the widest possible gap on both sides. Picture a road between two villages: an SVM builds the widest road it can without running over houses. That simple idea, plus a clever trick called the kernel, makes SVMs strong classifiers for small and medium-sized datasets, especially with many features.

In this lesson you will train linear and non-linear SVMs, understand the C and gamma parameters, see why scaling is essential, and get probabilities the modern way.

Margins and support vectors

For two classes that can be separated by a straight line, many lines would work. An SVM picks the one with the largest margin: the distance to the nearest points of each class. Those nearest points are the support vectors. They alone define the boundary; you could delete every other training point and get the same model.

Here a district cricket academy in Rohtak records practice hours and matches played for eight young players, and whether they were selected.

import numpy as np
from sklearn.svm import SVC

# [practice hours per week, matches played]; 1 = selected
X = np.array([[2, 3], [3, 2], [4, 4], [3, 5], [8, 9], [9, 7], [7, 8], [10, 10]])
y = np.array([0, 0, 0, 0, 1, 1, 1, 1])

svm = SVC(kernel="linear", C=1.0).fit(X, y)
print("Support vectors:", svm.support_vectors_.tolist())
w, b = svm.coef_[0], svm.intercept_[0]
print("w =", w.round(3), " b =", round(b, 3))
print("Margin width:", round(2 / np.linalg.norm(w), 3))

new = [[5, 5], [6, 7]]
print("Predictions:", svm.predict(new))
print("Decision values:", svm.decision_function(new).round(2))

Support vectors: [[4.0, 4.0], [3.0, 5.0], [7.0, 8.0]]
w = [0.286 0.286]  b = -3.285
Margin width: 4.952
Predictions: [0 1]
Decision values: [-0.43  0.43]

Only three of the eight players are support vectors. The decision function is the signed distance-like score w·x + b: negative means class 0, positive means class 1, and values between -1 and 1 fall inside the margin. Both new players are inside it, which tells you they are borderline cases.

Soft margins and the C parameter

Real data overlaps, so a perfect separating line rarely exists. A soft-margin SVM allows some points inside the margin or on the wrong side, and pays a penalty for each. C sets how expensive those violations are:

• Large C: violations are costly, so the SVM squeezes the margin to classify training points correctly. Risk: overfitting.
• Small C: violations are cheap, so the SVM prefers a wide, simple margin, even if some training points end up on the wrong side. Risk: underfitting.

from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

X, y = make_blobs(n_samples=200, centers=[[0, 0], [3, 3]], cluster_std=1.5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

for C in [0.001, 0.01, 1, 100]:
    svm = SVC(kernel="linear", C=C).fit(X_train, y_train)
    print(f"C={C:<6} support vectors {svm.n_support_.sum():>3}   "
          f"train {svm.score(X_train, y_train):.3f}   test {svm.score(X_test, y_test):.3f}")

C=0.001  support vectors 124   train 0.936   test 0.933
C=0.01   support vectors  58   train 0.914   test 0.933
C=1      support vectors  25   train 0.936   test 0.933
C=100    support vectors  23   train 0.936   test 0.933

With a tiny C, the margin is so wide that most points sit inside it and become support vectors. As C grows, the margin narrows and fewer points define it. On this data the test score hardly changes, but on other data the choice matters a lot, so find the best C with cross-validation. The default is C=1.0.

Kernels: curved boundaries

When the classes cannot be separated by a straight line, an SVM can use a kernel. A kernel measures the similarity between two points in a way that behaves as if the data had been mapped into a higher-dimensional space, where a flat boundary does separate the classes. The SVM never builds that space; it only uses the similarities, which keeps it fast. This is called the kernel trick.

For the RBF (radial basis function) kernel, gamma controls how far one training point's influence reaches. A small gamma gives smooth, broad boundaries; a large gamma lets every point bend the boundary around itself, which overfits. The default gamma="scale" picks a value from the number of features and the spread of the data, and is a good start.

from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

X, y = make_moons(n_samples=300, noise=0.2, random_state=42)   # two curved half-moons
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

for kernel in ["linear", "poly", "rbf"]:
    svm = SVC(kernel=kernel).fit(X_train, y_train)
    print(f"{kernel:<7} test accuracy {svm.score(X_test, y_test):.3f}")

for gamma in [0.1, 1, 10, 100]:
    svm = SVC(kernel="rbf", gamma=gamma).fit(X_train, y_train)
    print(f"rbf gamma={gamma:<4} train {svm.score(X_train, y_train):.3f}   test {svm.score(X_test, y_test):.3f}")

linear  test accuracy 0.900
poly    test accuracy 0.911
rbf     test accuracy 0.967
rbf gamma=0.1  train 0.838   test 0.889
rbf gamma=1    train 0.943   test 0.978
rbf gamma=10   train 0.962   test 0.978
rbf gamma=100  train 0.986   test 0.944

The RBF kernel follows the moon shapes best. With gamma=100, the training score is nearly perfect while the test score falls: classic overfitting. You can see the boundary with DecisionBoundaryDisplay:

import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from sklearn.inspection import DecisionBoundaryDisplay
from sklearn.svm import SVC

X, y = make_moons(n_samples=300, noise=0.2, random_state=42)
svm = SVC(kernel="rbf", C=1.0).fit(X, y)

disp = DecisionBoundaryDisplay.from_estimator(svm, X, response_method="predict", alpha=0.3)
disp.ax_.scatter(X[:, 0], X[:, 1], c=y, edgecolor="k", s=20)
disp.ax_.set_title("RBF SVM on the moons data")
plt.show()

Scaling, probabilities and SVMs in practice

SVMs measure distances, so features must be on similar scales. Without scaling, one large feature dominates, just as with k-nearest neighbours.

SVC does not produce probabilities directly. Its decision_function gives scores, which are all you need for ranking metrics such as ROC-AUC. When you need real probabilities, wrap the SVM in CalibratedClassifierCV, which learns how to turn the scores into probabilities.

from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

raw = SVC().fit(X_train, y_train)
scaled = make_pipeline(StandardScaler(), SVC()).fit(X_train, y_train)
print(f"Without scaling: {raw.score(X_test, y_test):.3f}")
print(f"With scaling:    {scaled.score(X_test, y_test):.3f}")
print(f"ROC-AUC from decision_function: {roc_auc_score(y_test, scaled.decision_function(X_test)):.3f}")

calibrated = CalibratedClassifierCV(make_pipeline(StandardScaler(), SVC()), ensemble=False)
calibrated.fit(X_train, y_train)
print("P(benign) for 3 patients:", calibrated.predict_proba(X_test[:3])[:, 1].round(3))

Without scaling: 0.923
With scaling:    0.979
ROC-AUC from decision_function: 0.997
P(benign) for 3 patients: [0.988 0.    0.581]

A few more members of the family:

• SVR does regression. It fits a tube of width epsilon around the data and ignores errors inside the tube.
• LinearSVC and LinearSVR only support linear boundaries but use a much faster solver, so they suit large datasets and text data with thousands of features.
• Kernel SVMs get slow as data grows: training time rises much faster than the number of rows. Beyond tens of thousands of rows, prefer LinearSVC, logistic regression or gradient boosting.

Common mistakes

• Forgetting to scale. Always put a StandardScaler before an SVM in a pipeline.
• Tuning C and gamma one at a time by hand. They interact; search them together with cross-validation, as in the exercise below.
• Using SVC(probability=True) in new code. It is deprecated in 1.9; use CalibratedClassifierCV.
• Using kernel SVMs on very large datasets. Training becomes very slow; use a linear model or boosting.

Key points

• An SVM finds the boundary with the widest margin; the closest points, the support vectors, define it.
• C trades margin width against training errors: large C fits tighter, small C keeps a wider margin.
• Kernels give curved boundaries; RBF is the default, and gamma controls how wiggly it can be.
• Scale features before an SVM.
• Use decision_function for ROC-AUC and CalibratedClassifierCV when you need probabilities.

from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_wine(return_X_y=True)
results = {}
for C in [0.1, 1, 10]:
    for gamma in ["scale", 0.01, 0.1]:
        model = make_pipeline(StandardScaler(), SVC(C=C, gamma=gamma))
        results[(C, gamma)] = cross_val_score(model, X, y, cv=5).mean()
        print(f"C={C:<4} gamma={gamma!s:<6} {results[(C, gamma)]:.3f}")
print("Best:", max(results, key=results.get))

C=0.1  gamma=scale  0.966
C=0.1  gamma=0.01   0.748
C=0.1  gamma=0.1    0.955
C=1    gamma=scale  0.983
C=1    gamma=0.01   0.972
C=1    gamma=0.1    0.978
C=10   gamma=scale  0.989
C=10   gamma=0.01   0.978
C=10   gamma=0.1    0.989
Best: (10, 'scale')