Logistic Regression

Despite its name, logistic regression is a classification model. It answers yes/no questions such as "Is this tumour malignant?" or "Will this customer cancel?", and it gives a probability with every answer. It is fast, easy to explain and often surprisingly strong, so it is the first classifier most data scientists try.

In this lesson you will see how logistic regression turns a straight line into a probability, use probabilities and thresholds, handle more than two classes, and control the model with regularisation.

From a line to a probability

Linear regression produces any number, from minus infinity to plus infinity. A probability must stay between 0 and 1. Logistic regression first computes a linear score z = w·x + b, exactly like linear regression, and then squeezes it through the sigmoid (logistic) function:

p = 1 / (1 + e^(-z))

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

for z in [-6, -2, -1, 0, 1, 2, 6]:
    print(f"z = {z:>2} -> probability {sigmoid(z):.3f}")

z = -6 -> probability 0.002
z = -2 -> probability 0.119
z = -1 -> probability 0.269
z =  0 -> probability 0.500
z =  1 -> probability 0.731
z =  2 -> probability 0.881
z =  6 -> probability 0.998

A score of 0 means 50%. Large positive scores approach 1, large negative scores approach 0, and the curve between them is a smooth S shape. Training finds the weights w and bias b that make the predicted probabilities match the real labels as closely as possible (it minimises a quantity called log loss).

A first binary classifier

Twelve students recorded their study hours, and whether they passed. Notice the overlap: some students who studied 3 or 4 hours failed and some passed. Logistic regression handles this naturally by giving probabilities instead of hard rules.

import numpy as np
from sklearn.linear_model import LogisticRegression

hours = np.array([[0.5], [1], [1.5], [2], [2.5], [3], [3.5], [4], [4.5], [5], [5.5], [6]])
passed = np.array([0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1])

model = LogisticRegression().fit(hours, passed)
w, b = model.coef_[0][0], model.intercept_[0]
print(f"w = {w:.2f}, b = {b:.2f}, 50% point at {-b / w:.2f} hours")

for h in [1, 3, 5]:
    p = model.predict_proba([[h]])[0, 1]
    print(f"{h} hours -> P(pass) = {p:.2f}, predicted class {model.predict([[h]])[0]}")

w = 1.01, b = -3.28, 50% point at 3.25 hours
1 hours -> P(pass) = 0.09, predicted class 0
3 hours -> P(pass) = 0.44, predicted class 0
5 hours -> P(pass) = 0.85, predicted class 1

predict_proba returns one column per class, in the order of model.classes_; column 1 is the probability of class 1 (pass). predict simply applies a threshold of 0.5: probability 0.5 or more means class 1. The 50% point is where w·x + b = 0, so at -b / w hours.

Probabilities and thresholds on real data

The 0.5 threshold is only a default. In cancer screening, missing a malignant tumour is much worse than a false alarm that leads to one more test. Lowering the threshold catches more malignant cases, at the cost of more false alarms. Here, recall is the share of malignant tumours that were caught, and precision is the share of alarms that were correct; Classification Metrics covers both in detail.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y  # 1 = malignant
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]
print("P(malignant) for 5 patients:", proba[:5].round(3))

for threshold in [0.5, 0.3, 0.1]:
    pred = (proba >= threshold).astype(int)
    print(f"threshold {threshold}: caught {recall_score(y_test, pred):.3f} of malignant, "
          f"alarms correct {precision_score(y_test, pred):.3f}")

P(malignant) for 5 patients: [0.999 0.004 1.    0.    0.035]
threshold 0.5: caught 0.925 of malignant, alarms correct 0.980
threshold 0.3: caught 0.962 of malignant, alarms correct 0.981
threshold 0.1: caught 0.981 of malignant, alarms correct 0.897

At a threshold of 0.1, 98% of malignant tumours are caught (up from 92.5% at 0.5), but about one alarm in ten is now a false one. Which threshold is right is a business or medical decision, not a technical one. Logistic regression is regularised and uses gradient-based solvers, so scale the features (here with StandardScaler in a pipeline); unscaled data often triggers a ConvergenceWarning.

More than two classes

For three or more classes, scikit-learn's logistic regression learns one set of weights per class and turns the scores into probabilities that add up to 1, using the softmax function (the multi-class version of the sigmoid). This is called multinomial logistic regression and happens automatically.

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression

iris = load_iris()
model = LogisticRegression(max_iter=1000).fit(iris.data, iris.target)

flower = [[6.0, 2.9, 4.5, 1.5]]
proba = model.predict_proba(flower)[0]
for name, p in zip(iris.target_names, proba):
    print(f"{name:<11} {p:.3f}")
print("Sum:", round(proba.sum(), 3), "| prediction:", iris.target_names[model.predict(flower)[0]])
print("coef_ shape:", model.coef_.shape)

setosa      0.005
versicolor  0.813
virginica   0.182
Sum: 1.0 | prediction: versicolor
coef_ shape: (3, 4)

coef_ now has one row of four weights per class. The prediction is the class with the highest probability.

Regularisation with C

LogisticRegression is regularised by default, like Ridge from Polynomial Features & Regularisation. The strength is set with C, which is the inverse of the strength: small C means a strong penalty and a simpler model; large C means a weak penalty. The default is C=1.0.

The type of penalty is set with l1_ratio: 0 (the default) gives the L2 penalty that shrinks all weights, and 1 gives the L1 penalty that sets some weights exactly to zero. L1 needs the "liblinear" or "saga" solver.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

for C in [0.01, 0.1, 1, 10]:
    model = make_pipeline(StandardScaler(),
                          LogisticRegression(C=C, l1_ratio=1.0, solver="liblinear", random_state=42))
    model.fit(X_train, y_train)
    used = np.sum(model[-1].coef_ != 0)
    print(f"C={C:<5} test accuracy {model.score(X_test, y_test):.3f}, features used {used} of 30")

C=0.01  test accuracy 0.909, features used 4 of 30
C=0.1   test accuracy 0.972, features used 7 of 30
C=1     test accuracy 0.993, features used 16 of 30
C=10    test accuracy 0.958, features used 22 of 30

With C=0.01 the penalty is so strong that only 4 of the 30 measurements keep a non-zero weight, yet accuracy is still 91%. C=1 gives the best test score here, and C=10 uses more features but scores lower, a sign that the weak penalty lets the model start to overfit. Choosing C well is a job for cross-validation, for example with LogisticRegressionCV or a grid search (see Cross-Validation & Hyperparameter Tuning).

Common mistakes

• Using logistic regression to predict a number. It is a classifier; for numbers use Linear Regression.
• Ignoring a ConvergenceWarning. Scale the features first; if the warning remains, raise max_iter.
• Treating 0.5 as the only threshold. Pick the threshold from the cost of each kind of mistake.
• Reading predict_proba columns in the wrong order. Columns follow model.classes_, not the order you expect.

Key points

• Logistic regression computes a linear score and turns it into a probability with the sigmoid (or softmax for several classes).
• predict_proba gives probabilities; predict applies a 0.5 threshold, which you can change.
• Lowering the threshold catches more positives (higher recall) at the cost of more false alarms.
• Regularisation is on by default; smaller C means a stronger penalty.
• In scikit-learn 1.8, choose L1 or L2 with l1_ratio instead of the deprecated penalty parameter.

from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
print(f"Test accuracy: {model.score(X_test, y_test):.3f}")
print("coef_ shape:", model[-1].coef_.shape)
print("First wine probabilities:", model.predict_proba(X_test[:1])[0].round(3), "true class:", y_test[0])

Test accuracy: 1.000
coef_ shape: (3, 13)
First wine probabilities: [0.926 0.072 0.002] true class: 0