Project: Salary Estimator (Machine Learning Tutorial)

In this project you will build a complete machine learning system from a raw CSV file to a saved model that answers real questions. It brings together almost everything in this course: pandas cleaning, train/test splits, pipelines, cross-validation, tuning, regression metrics, responsible AI checks and saving models.

The brief. The HR team of an IT services company with offices in Bengaluru, Hyderabad, Chennai, Pune and Gurugram wants a tool that suggests a fair annual salary (CTC, in rupees) for a role, based on a person's experience, education, department, city and work mode. Recruiters will use it as a second opinion when preparing offers, and HR will use it to spot existing salaries that look out of line. The data is employees.csv: 302 employee records, available in the browser editor for every example below.

Step 1: Frame the problem

Before touching the data, write down the decisions that shape everything else:

Type of problem: the target, salary, is a number, so this is regression.
Metric: mean absolute error (MAE) in rupees, because "the estimate is usually within ₹70,000" is easy for HR to understand. We will also report R² and the percentage error (see Regression Metrics).
Success: clearly beat a baseline that always predicts the average salary, with errors small enough to be useful for offers.
Allowed features: only information that is known when making an offer and that may fairly influence pay. Name and employee ID identify people and carry no useful signal. Gender and age must not drive pay decisions (and age mostly repeats experience). Performance rating and whether someone left the company are not known for a new candidate. So we use experience, education, department, city and work mode, and later check the model's errors by gender instead of using gender as an input.

Writing these choices down first is what separates a project from an experiment: it makes the result checkable.

Step 2: Load and explore

Start by looking, not modelling.

PythonRun it yourself

import pandas as pd

df = pd.read_csv("employees.csv")
print("Shape:", df.shape)
print(df[["department", "city", "education", "experience_years", "work_mode", "salary"]].head())
print("Missing values:", df.isna().sum()[df.isna().sum() > 0].to_dict())
print("Duplicate rows:", df.duplicated().sum())
print(df["department"].value_counts().to_dict())

Output

Shape: (302, 13)
    department       city education  experience_years work_mode     salary
0  Engineering  Hyderabad    B.Tech                 3    Office   705000.0
1      Support    Chennai     B.Com                 3    Remote   387000.0
2  Engineering  Hyderabad    M.Tech                12    Hybrid  1791000.0
3      Support  Bengaluru    B.Tech                 3    Office   534000.0
4      Support  Bengaluru    B.Tech                 7    Hybrid   758000.0
Missing values: {'salary': 5}
Duplicate rows: 2
{'Engineering': 97, 'Sales': 60, 'Support': 49, 'Data Science': 38, 'Finance': 29, 'HR': 23, 'hr': 2, 'support': 2, 'finance': 1, 'engineering': 1}

Three problems stand out:

two rows are exact duplicates;
five employees have no salary recorded;
department names are inconsistent: "HR" and "hr", "Support" and "support", and so on.

Step 3: Clean the data

Fix each problem with pandas, as in Preparing Data with pandas. Rows without a salary cannot be used for training, because the target is what we learn from, so we drop them rather than invent values.

PythonRun it yourself

import pandas as pd

df = pd.read_csv("employees.csv")
df = df.drop_duplicates()
df["department"] = df["department"].str.strip().str.title().replace({"Hr": "HR"})
df = df.dropna(subset=["salary"]).reset_index(drop=True)

print("Rows after cleaning:", len(df))
print(df["department"].value_counts().to_dict())
print(df["salary"].describe().round(0))
print(df.groupby("education")["salary"].median().sort_values().to_dict())

Output

Rows after cleaning: 295
{'Engineering': 94, 'Sales': 59, 'Support': 51, 'Data Science': 38, 'Finance': 29, 'HR': 24}
count        295.0
mean      885641.0
std       328669.0
min       337000.0
25%       650000.0
50%       819000.0
75%      1046000.0
max      1979000.0
Name: salary, dtype: float64
{'B.Com': 612000.0, 'B.Sc': 657000.0, 'B.Tech': 846500.0, 'MBA': 848000.0, 'MCA': 979000.0, 'M.Tech': 1183000.0}

The cleaned data has one spelling per department. Salaries run from a few lakh to around twenty lakh a year, and the median clearly rises with education level. Note the .title() trap: it turns "HR" into "Hr", so we map it back.

Step 4: Split and set a baseline

Split once, keep the test set locked away, and measure the simplest possible model. Every later step must beat this number.

PythonRun it yourself

import pandas as pd
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

df = pd.read_csv("employees.csv").drop_duplicates()
df["department"] = df["department"].str.strip().str.title().replace({"Hr": "HR"})
df = df.dropna(subset=["salary"]).reset_index(drop=True)

FEATURES = ["experience_years", "education", "department", "city", "work_mode"]
X_train, X_test, y_train, y_test = train_test_split(
    df[FEATURES], df["salary"], test_size=0.2, random_state=42)
print("Training rows:", len(X_train), " test rows:", len(X_test))

baseline = DummyRegressor(strategy="mean").fit(X_train, y_train)
print(f"Baseline MAE on the training set: ₹{mean_absolute_error(y_train, baseline.predict(X_train)):,.0f}")

Output

Training rows: 236  test rows: 59
Baseline MAE on the training set: ₹246,900

We report the baseline on the training data here, because the test set stays untouched until Step 8. An estimate that is typically off by more than ₹2 lakh is useless for offers.

Step 5: Build a pipeline and compare models

The features mix one number with four text categories, so a ColumnTransformer scales the number and one-hot encodes the categories. We wrap it with three different models and compare them with 5-fold cross-validation on the training set only.

PythonRun it yourself

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingRegressor, RandomForestRegressor
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.read_csv("employees.csv").drop_duplicates()
df["department"] = df["department"].str.strip().str.title().replace({"Hr": "HR"})
df = df.dropna(subset=["salary"]).reset_index(drop=True)
FEATURES = ["experience_years", "education", "department", "city", "work_mode"]
X_train, X_test, y_train, y_test = train_test_split(
    df[FEATURES], df["salary"], test_size=0.2, random_state=42)

def preprocess():
    return ColumnTransformer([
        ("num", StandardScaler(), ["experience_years"]),
        ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), ["education", "department", "city", "work_mode"]),
    ])

cv = KFold(n_splits=5, shuffle=True, random_state=42)
models = {"Ridge": Ridge(alpha=1.0),
          "Random forest": RandomForestRegressor(n_estimators=100, random_state=42),
          "Hist gradient boosting": HistGradientBoostingRegressor(random_state=42)}
for name, model in models.items():
    mae = -cross_val_score(make_pipeline(preprocess(), model), X_train, y_train,
                           cv=cv, scoring="neg_mean_absolute_error").mean()
    print(f"{name:<23} CV MAE ₹{mae:,.0f}")

Output

Ridge                   CV MAE ₹76,456
Random forest           CV MAE ₹86,163
Hist gradient boosting  CV MAE ₹87,064

Ridge, the simplest model, wins clearly, and all three beat the baseline by a wide margin. With only 236 training rows and a mostly additive pattern (an extra year of experience, a higher degree or a different department each shift pay in a regular way), a linear model fits well, while the tree ensembles would need much more data to learn the same smooth shape. We keep Ridge and try to make it better.

Step 6: Engineer the target and tune

Salaries tend to grow by percentages: a promotion or an M.Tech adds a share of pay rather than a fixed amount. A common trick is to model the logarithm of the salary, which turns percentage effects into additive ones that a linear model handles well. TransformedTargetRegressor applies np.log before fitting and np.exp to every prediction automatically, so we still get rupees out. At the same time, we tune Ridge's alpha with GridSearchCV.

PythonRun it yourself

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer, TransformedTargetRegressor
from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV, KFold, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.read_csv("employees.csv").drop_duplicates()
df["department"] = df["department"].str.strip().str.title().replace({"Hr": "HR"})
df = df.dropna(subset=["salary"]).reset_index(drop=True)
FEATURES = ["experience_years", "education", "department", "city", "work_mode"]
X_train, X_test, y_train, y_test = train_test_split(
    df[FEATURES], df["salary"], test_size=0.2, random_state=42)

preprocess = ColumnTransformer([
    ("num", StandardScaler(), ["experience_years"]),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), ["education", "department", "city", "work_mode"]),
])
cv = KFold(n_splits=5, shuffle=True, random_state=42)

for name, target_log in [("salary as is", False), ("log of salary", True)]:
    ridge = make_pipeline(preprocess, Ridge())
    model = TransformedTargetRegressor(ridge, func=np.log, inverse_func=np.exp) if target_log else ridge
    prefix = "regressor__ridge__alpha" if target_log else "ridge__alpha"
    search = GridSearchCV(model, {prefix: [0.01, 0.1, 1, 10]}, cv=cv, scoring="neg_mean_absolute_error")
    search.fit(X_train, y_train)
    print(f"{name:<14} best alpha {search.best_params_[prefix]:<4} CV MAE ₹{-search.best_score_:,.0f}")

Output

salary as is   best alpha 0.1  CV MAE ₹76,361
log of salary  best alpha 0.1  CV MAE ₹64,071

Modelling the logarithm cuts the error from about ₹76,000 to about ₹64,000, a bigger gain than any switch of model gave. The best alpha (0.1) lies inside the grid we searched, so there is no need to search further out. Our final choice: Ridge with alpha=0.1 on the log of salary. (Inside TransformedTargetRegressor, the pipeline's parameters get the extra prefix regressor__.)

Step 7: Check the remaining errors

Before the final test, look at where the chosen model goes wrong, still using only training data. cross_val_predict gives an honest out-of-fold prediction for every training row.

PythonRun it yourself

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer, TransformedTargetRegressor
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold, cross_val_predict, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.read_csv("employees.csv").drop_duplicates()
df["department"] = df["department"].str.strip().str.title().replace({"Hr": "HR"})
df = df.dropna(subset=["salary"]).reset_index(drop=True)
FEATURES = ["experience_years", "education", "department", "city", "work_mode"]
train, test = train_test_split(df, test_size=0.2, random_state=42)

preprocess = ColumnTransformer([
    ("num", StandardScaler(), ["experience_years"]),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), ["education", "department", "city", "work_mode"]),
])
model = TransformedTargetRegressor(make_pipeline(preprocess, Ridge(alpha=0.1)),
                                   func=np.log, inverse_func=np.exp)
cv = KFold(n_splits=5, shuffle=True, random_state=42)
pred = cross_val_predict(model, train[FEATURES], train["salary"], cv=cv)
errors = (train["salary"] - pred).abs()

print("Mean absolute error by department:")
print(errors.groupby(train["department"]).mean().round(-3).sort_values().to_dict())
print("Mean absolute error by gender (gender was NOT a model input):")
print(errors.groupby(train["gender"]).mean().round(-3).to_dict())

Output

Mean absolute error by department:
{'Support': 37000.0, 'Sales': 50000.0, 'HR': 57000.0, 'Finance': 64000.0, 'Engineering': 73000.0, 'Data Science': 96000.0}
Mean absolute error by gender (gender was NOT a model input):
{'F': 66000.0, 'M': 62000.0}

Errors are smallest in Support, where salaries are lowest, and largest in Data Science, where salaries are highest and most varied. The average error for women and men differs by only a few thousand rupees, so the model is not clearly more wrong for one gender. This check has limits: a model trained on biased past salaries would repeat that bias for everyone, accurately. HR should also compare actual pay between women and men with the same profile.

Step 8: Final test, save and predict

Now, once, we fit the chosen model on all the training data and score it on the untouched test set. Then we save the whole pipeline with joblib, along with metadata, and use the saved file to suggest salaries for three new candidates.

PythonRun it yourself

import json
import joblib
import numpy as np
import pandas as pd
import sklearn
from sklearn.compose import ColumnTransformer, TransformedTargetRegressor
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error, mean_absolute_percentage_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.read_csv("employees.csv").drop_duplicates()
df["department"] = df["department"].str.strip().str.title().replace({"Hr": "HR"})
df = df.dropna(subset=["salary"]).reset_index(drop=True)
FEATURES = ["experience_years", "education", "department", "city", "work_mode"]
X_train, X_test, y_train, y_test = train_test_split(
    df[FEATURES], df["salary"], test_size=0.2, random_state=42)

preprocess = ColumnTransformer([
    ("num", StandardScaler(), ["experience_years"]),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), ["education", "department", "city", "work_mode"]),
])
model = TransformedTargetRegressor(make_pipeline(preprocess, Ridge(alpha=0.1)),
                                   func=np.log, inverse_func=np.exp).fit(X_train, y_train)
pred = model.predict(X_test)
mae = mean_absolute_error(y_test, pred)
print(f"Test MAE ₹{mae:,.0f}   R² {r2_score(y_test, pred):.3f}   "
      f"MAPE {mean_absolute_percentage_error(y_test, pred):.1%}")

joblib.dump(model, "salary_model.joblib")
with open("salary_model.json", "w", encoding="utf-8") as f:
    json.dump({"features": FEATURES, "target": "annual salary (Rs)", "test_mae": round(mae),
               "sklearn_version": sklearn.__version__, "data_rows": len(df)}, f, indent=2)

loaded = joblib.load("salary_model.joblib")
candidates = pd.DataFrame([
    {"experience_years": 2, "education": "B.Tech", "department": "Engineering", "city": "Pune", "work_mode": "Hybrid"},
    {"experience_years": 6, "education": "M.Tech", "department": "Data Science", "city": "Bengaluru", "work_mode": "Office"},
    {"experience_years": 4, "education": "B.Com", "department": "Finance", "city": "Chennai", "work_mode": "Remote"},
])
for (_, person), estimate in zip(candidates.iterrows(), loaded.predict(candidates)):
    low, high = estimate - mae, estimate + mae
    print(f"{person['education']:<6} {person['department']:<12} {person['experience_years']} yrs: "
          f"about ₹{round(estimate, -4):,.0f} (typical range ₹{round(low, -4):,.0f} to ₹{round(high, -4):,.0f})")

Output

Test MAE ₹63,678   R² 0.940   MAPE 7.6%
B.Tech Engineering  2 yrs: about ₹770,000 (typical range ₹710,000 to ₹840,000)
M.Tech Data Science 6 yrs: about ₹1,490,000 (typical range ₹1,430,000 to ₹1,560,000)
B.Com  Finance      4 yrs: about ₹620,000 (typical range ₹560,000 to ₹690,000)

On the test set, which no earlier step touched, the typical error is about ₹64,000 (7.6%) with an R² of 0.94, in line with the cross-validated estimate, so the result is trustworthy. The baseline was off by about ₹2.5 lakh. The suggestions look sensible: the M.Tech data scientist with six years in Bengaluru gets the highest estimate. Rounding to ₹10,000 and showing a range reminds users that this is an estimate, not an exact answer.

Step 9: Document the model

A model that others will use needs a short model card. Ours could read:

Purpose: suggest a fair annual salary range for a role, as a second opinion for recruiters and HR. Not for automatic decisions.
Data: 295 cleaned employee records from one company (duplicates removed, rows without salary dropped).
Inputs: experience, education, department, city, work mode. Gender, age, names, ratings and attrition are deliberately excluded.
Model: Ridge regression on the logarithm of salary, inside a scikit-learn 1.8 pipeline.
Performance: test MAE about ₹64,000 (7.6%), R² 0.94; errors checked by department and gender.
Limits: learned from this company's past pay, so it repeats any unfairness in that pay; based on few rows for small departments; does not know about skills, certifications or the job market; must be retrained as salaries change.

Where to go next

You now have a working, documented, saved model. Ideas to extend it:

Wrap the saved pipeline in a small FastAPI service, as shown in MLOps Basics, and validate incoming requests.
Add a drift check that compares new offers with the training data each month.
Try HistGradientBoostingRegressor with tuned settings and see whether it ever beats Ridge as more data arrives.
Use the model the other way round: list current employees whose salary is far below the estimate and send those cases for a human pay review.

Common mistakes

Using features that will not exist at prediction time, such as performance ratings for a new candidate.
Letting protected characteristics drive pay, directly or through proxies; exclude them, and check errors by group instead.
Peeking at the test set while choosing models. All comparisons here used cross-validation on the training set.
Forgetting the cleaning steps at prediction time. Put every learned step in the pipeline, and apply the same text fixes to new input.

Key points

Start every project by fixing the target, the metric, the success bar and the allowed features.
Clean and explore with pandas before modelling, and keep the test set locked until the end.
A ColumnTransformer pipeline plus cross-validation makes model comparison fair and leak-free.
Simple ideas, such as modelling the logarithm of a skewed target, can beat more complex models.
Check errors by group, save the whole pipeline with metadata, and document the model's purpose and limits.

Exercise

Extend the project in two ways. First, add HistGradientBoostingRegressor(random_state=42) on the log of salary to the comparison and print its cross-validated MAE next to the log-Ridge model. Second, use the final model to find the five employees in the whole cleaned dataset whose actual salary is furthest below the model's estimate, and print their department, experience and the gap in rupees.

Show solution

Python

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer, TransformedTargetRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold, cross_val_predict, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.read_csv("employees.csv").drop_duplicates()
df["department"] = df["department"].str.strip().str.title().replace({"Hr": "HR"})
df = df.dropna(subset=["salary"]).reset_index(drop=True)
FEATURES = ["experience_years", "education", "department", "city", "work_mode"]
X_train, X_test, y_train, y_test = train_test_split(
    df[FEATURES], df["salary"], test_size=0.2, random_state=42)

def log_model(regressor):
    preprocess = ColumnTransformer([
        ("num", StandardScaler(), ["experience_years"]),
        ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), ["education", "department", "city", "work_mode"]),
    ])
    return TransformedTargetRegressor(make_pipeline(preprocess, regressor),
                                      func=np.log, inverse_func=np.exp)

cv = KFold(n_splits=5, shuffle=True, random_state=42)
for name, reg in [("Ridge", Ridge(alpha=0.1)),
                  ("HistGradientBoosting", HistGradientBoostingRegressor(random_state=42))]:
    mae = -cross_val_score(log_model(reg), X_train, y_train, cv=cv,
                           scoring="neg_mean_absolute_error").mean()
    print(f"{name:<21} CV MAE ₹{mae:,.0f}")

estimate = cross_val_predict(log_model(Ridge(alpha=0.1)), df[FEATURES], df["salary"], cv=cv)
df["gap"] = estimate - df["salary"]
top = df.nlargest(5, "gap")[["department", "experience_years", "salary", "gap"]]
print(top.round({"salary": -3, "gap": -3}).to_string(index=False))

Output

Ridge                 CV MAE ₹64,071
HistGradientBoosting  CV MAE ₹81,126
  department  experience_years    salary      gap
       Sales                21 1520000.0 435000.0
 Engineering                18 1849000.0 320000.0
 Engineering                14 1491000.0 289000.0
Data Science                14 1908000.0 288000.0
Data Science                 2  848000.0 240000.0

Gradient boosting on the log target improves on its Step 5 score but stays well behind Ridge. The second list gives HR a short review list, mostly experienced staff, such as a Sales employee with 21 years of experience earning about ₹4.4 lakh less than the estimate. The model cannot see everything (skills, role level, part-time work), so these are cases for a person to look into, not automatic pay changes.

Quick Quiz

1. Why does the project exclude performance rating from the features?

2. Why are rows with a missing salary dropped instead of filled?

3. What does TransformedTargetRegressor(..., func=np.log, inverse_func=np.exp) do?

4. How does the project check fairness without using gender as an input?

Web Development

Programming Languages

Databases

Data Science & AI

Cloud & DevOps

Security & Systems

Office & Productivity

Project: Salary Estimator

Step 1: Frame the problem

Step 2: Load and explore

Step 3: Clean the data

Step 4: Split and set a baseline

Step 5: Build a pipeline and compare models

Step 6: Engineer the target and tune

Step 7: Check the remaining errors

Step 8: Final test, save and predict

Step 9: Document the model

Where to go next

Common mistakes

Key points

Exercise

Quick Quiz

Learn Machine Learning with a mentor