Feature Engineering

A model can only learn from the columns you give it. If delivery time depends on the distance between the pickup and drop points, but your table only has their map coordinates, a simple model may never work out the distance. Add one well-chosen column and the same model suddenly works much better.

Feature engineering means creating, transforming and selecting features so that patterns become easy to learn. It is where domain knowledge (what you know about loans, shops or cricket) pays off most. In this lesson you will build features from ratios and dates, transform skewed numbers, encode categories with many values, and select the most useful features.

Features from domain knowledge

The most powerful features come from understanding the problem:

• Ratios and differences: EMI-to-income ratio, price per square foot, marks gained since the last exam.
• Date and time parts: hour of day, day of week, weekend or not, month, days since the last purchase.
• Counts and aggregates: number of orders in the last 30 days, average bill per customer.

pandas makes these easy. Here are orders from a Jaipur snack shop's app.

import pandas as pd

orders = pd.DataFrame({
    "placed_at": pd.to_datetime(["2026-10-02 19:45", "2026-10-03 13:10",
                                 "2026-10-04 21:30", "2026-10-06 08:05"]),
    "amount": [450, 1299, 760, 180],
    "items": [3, 5, 4, 1],
})
orders["hour"] = orders["placed_at"].dt.hour
orders["day"] = orders["placed_at"].dt.day_name()
orders["is_weekend"] = orders["placed_at"].dt.dayofweek >= 5
orders["amount_per_item"] = (orders["amount"] / orders["items"]).round(1)
print(orders.drop(columns="placed_at"))

   amount  items  hour       day  is_weekend  amount_per_item
0     450      3    19    Friday       False            150.0
1    1299      5    13  Saturday        True            259.8
2     760      4    21    Sunday        True            190.0
3     180      1     8   Tuesday       False            180.0

A raw timestamp is useless to most models, but "Sunday evening" may say a lot about order size. dayofweek counts Monday as 0, so values 5 and 6 are the weekend.

A good feature beats a fancier model

Let us test the delivery idea. A food delivery app in Pune records each order's pickup and drop points as coordinates on a city grid (kilometres east and north), and the delivery time in minutes. Time really depends on the distance between the two points. A linear model only adds up weighted columns, so it cannot square, add and take a square root of coordinates by itself.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(42)
n = 1000
pickup = rng.uniform(0, 20, size=(n, 2))     # km east, km north
drop = rng.uniform(0, 20, size=(n, 2))
distance = np.sqrt(((pickup - drop) ** 2).sum(axis=1))
minutes = 10 + 3 * distance + rng.normal(0, 4, n)

coordinates = np.hstack([pickup, drop])                  # 4 raw columns
with_distance = np.column_stack([coordinates, distance])  # plus 1 engineered column
for name, X in [("coordinates only", coordinates), ("coordinates + distance", with_distance)]:
    r2 = cross_val_score(LinearRegression(), X, minutes, cv=5, scoring="r2").mean()
    print(f"{name:<23} R² {r2:.3f}")

coordinates only        R² -0.006
coordinates + distance  R² 0.934

With only the coordinates, the model explains almost nothing; one engineered column changes everything. A flexible model such as gradient boosting could approximate the distance by itself, but it would need much more data and would be harder to explain. Domain knowledge ("time depends on distance") did the heavy lifting.

Transforming numeric features

Some transformations help many models:

• Log transform: incomes, prices and populations are skewed (a few huge values). np.log1p (log of 1 + x) compresses the large values so they do not dominate.
• Binning: cut a number into ranges such as low, middle and high income. KBinsDiscretizer does this; it can help linear models capture non-linear effects, at the cost of some detail.
• Interactions: products of features, such as area × floor, from PolynomialFeatures(interaction_only=True), as seen in Polynomial Features & Regularisation.

FunctionTransformer wraps any function as a scikit-learn transformer, so it can live in a pipeline.

import numpy as np
from sklearn.preprocessing import FunctionTransformer, KBinsDiscretizer

incomes = np.array([[18_000], [25_000], [32_000], [45_000], [60_000], [90_000], [250_000], [1_200_000]])

log_income = FunctionTransformer(np.log1p).fit_transform(incomes)
print("log1p:", log_income.ravel().round(2))

binner = KBinsDiscretizer(n_bins=3, encode="ordinal", strategy="quantile",
                          quantile_method="averaged_inverted_cdf")
print("bins: ", binner.fit_transform(incomes).ravel())
print("edges:", binner.bin_edges_[0].round(0))

log1p: [ 9.8  10.13 10.37 10.71 11.   11.41 12.43 14.  ]
bins:  [0. 0. 1. 1. 1. 2. 2. 2.]
edges: [  18000.   32000.   90000. 1200000.]

After the log, the gap between ₹18,000 and ₹12 lakh shrinks from a factor of 66 to a few units. The quantile bins put roughly the same number of people in each group. (Setting quantile_method explicitly avoids a warning about a default that changes in a future version.)

Categories with many values: TargetEncoder

One-hot encoding (see Scaling & Encoding Features) creates one column per category. With 500 cities or 20,000 PIN codes, that is too many. TargetEncoder (added in scikit-learn 1.3) replaces each category with a number based on the average target for that category, for example the churn rate of each city. It smooths rare categories towards the overall average, and during fit_transform it uses internal cross-fitting so that a row's own target never leaks into its own encoding.

import numpy as np
import pandas as pd
from sklearn.preprocessing import TargetEncoder

rng = np.random.default_rng(42)
cities = rng.choice(["Pune", "Jaipur", "Kochi", "Lucknow", "Rohtak", "Indore"], size=300)
true_rate = {"Pune": 0.3, "Jaipur": 0.15, "Kochi": 0.1, "Lucknow": 0.25, "Rohtak": 0.4, "Indore": 0.2}
churned = (rng.random(300) < np.array([true_rate[c] for c in cities])).astype(int)

encoder = TargetEncoder(random_state=42)
X_encoded = encoder.fit_transform(pd.DataFrame({"city": cities}), churned)
print(pd.Series(encoder.encodings_[0], index=encoder.categories_[0]).round(3))
print("First 4 training rows:", cities[:4], X_encoded[:4].ravel().round(3))

Indore     0.202
Jaipur     0.273
Kochi      0.098
Lucknow    0.318
Pune       0.305
Rohtak     0.410
dtype: float64
First 4 training rows: ['Pune' 'Rohtak' 'Lucknow' 'Kochi'] [0.317 0.416 0.323 0.062]

The learned encodings follow the true churn rates we used (Rohtak highest, Kochi lowest), with some noise because each city has only about 50 customers. The training rows get slightly different values, because each one was encoded by a model fitted on the other folds; new data passed to transform uses encodings_. In scikit-learn 1.9 the shuffle and random_state parameters are deprecated: you pass a splitter such as cv=KFold(5, shuffle=True, random_state=42) instead. In 1.8, cv must be a whole number, so use random_state as shown.

Feature selection

More features are not always better. Useless columns add noise, slow training and invite overfitting. Feature selection keeps the most informative ones:

• SelectKBest scores each feature on its own (for example with the ANOVA F-test f_classif or mutual_info_classif) and keeps the best k.
• SelectFromModel keeps the features a model finds important, such as non-zero Lasso coefficients or high forest importances.
• RFE (recursive feature elimination) repeatedly drops the weakest feature and refits.

Here we hide 20 columns of pure noise among the 30 real breast cancer features and see whether SelectKBest can find the real ones.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
rng = np.random.default_rng(42)
X_noisy = np.hstack([X, rng.normal(size=(len(X), 20))])   # columns 30-49 are noise

selector = SelectKBest(f_classif, k=10).fit(X_noisy, y)
chosen = np.flatnonzero(selector.get_support())
print("Chosen columns:", chosen)
print("Noise columns chosen:", np.sum(chosen >= 30))

for name, model in [("all 50 columns", make_pipeline(StandardScaler(), LogisticRegression())),
                    ("best 10 columns", make_pipeline(StandardScaler(), SelectKBest(f_classif, k=10),
                                                      LogisticRegression()))]:
    print(f"{name:<16} accuracy {cross_val_score(model, X_noisy, y, cv=5).mean():.3f}")

Chosen columns: [ 0  2  3  6  7 20 22 23 26 27]
Noise columns chosen: 0
all 50 columns   accuracy 0.965
best 10 columns  accuracy 0.951

The selector ignores every noise column. The 10 chosen features score slightly below all 50 columns here: logistic regression copes well with a few noise columns, and 10 is not necessarily the best number to keep. Putting SelectKBest inside the pipeline matters: the selection is then redone on each training fold, so the test folds never influence which features are chosen.

Common mistakes

• Leaking the answer into a feature. "Days since loan closed" when predicting loan default only exists after the outcome; every feature must be known at prediction time.
• Computing statistics on all rows before splitting. Target encodings, bin edges and feature selection must be learned from training data only; pipelines do this for you.
• Forgetting to repeat the steps at prediction time. Wrap transformations in FunctionTransformer or a pipeline so new data gets exactly the same features.
• Adding hundreds of features blindly. Each one should have a reason; check that it helps with cross-validation.

Key points

• Feature engineering turns raw data into columns that make patterns easy to learn.
• Ratios, differences, date parts and aggregates from domain knowledge are often the most valuable features.
• Use np.log1p for skewed numbers, KBinsDiscretizer for ranges and PolynomialFeatures for interactions.
• TargetEncoder handles categories with many values by encoding each one with a smoothed target average.
• Select features with SelectKBest, SelectFromModel or RFE, inside a pipeline to avoid leakage.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
rng = np.random.default_rng(42)
X_noisy = np.hstack([X, rng.normal(size=(len(X), 20))])

for k in [2, 5, 10, 20, 50]:
    model = make_pipeline(StandardScaler(), SelectKBest(f_classif, k=k), LogisticRegression())
    print(f"k={k:<3} accuracy {cross_val_score(model, X_noisy, y, cv=5).mean():.3f}")

k=2   accuracy 0.935
k=5   accuracy 0.949
k=10  accuracy 0.951
k=20  accuracy 0.979
k=50  accuracy 0.965