An EDA Checklist

Exploratory data analysis (EDA) is the first look at a dataset, before any model or report. Beginners often do it as a random walk: a describe() here, a chart there, and whatever looked interesting. Experienced analysts follow a checklist, because the costly mistakes in EDA are not the charts you draw but the checks you forget: duplicate IDs, a column that is mostly empty, a "pattern" based on ten people.

This lesson turns EDA into a checklist you can reuse on any table. You will apply it to employees.csv, a practice HR dataset, with one question in mind: why do employees leave? The Project: Sales Data EDA showed one complete analysis; this lesson gives you the general method behind it.

The checklist

1. Question and unit. Write the question in one sentence and say what one row is (one employee, one order, one day).
2. Structure. Rows, columns, dtypes, duplicate rows and duplicate IDs.
3. Quality. Missing values, impossible values and inconsistent spellings for every column.
4. One column at a time. Ranges, skew, rare categories, the share of each category.
5. Missingness. Is anything missing in a pattern, or at random?
6. Segments. The key measure broken down by each important category, always with group sizes.
7. Time. Does the picture change by period?
8. Relationships. Correlations between numeric columns, read with care.
9. Log. Write findings, caveats and next questions as you go.

Here the question is "Which employees leave, and what is different about them?", the unit is one employee, and the key measure is the attrition rate: the share of employees with left == "Yes".

Structure and quality in one table

A small profile() function answers steps 2 and 3 for every column at once. Keep it in your toolbox and run it on every new table.

import pandas as pd

emp = pd.read_csv("employees.csv", parse_dates=["join_date"])

def profile(df):
    rows = []
    for col in df.columns:
        s = df[col]
        rows.append({
            "column": col,
            "dtype": str(s.dtype),
            "missing_%": round(s.isna().mean() * 100, 1),
            "unique": s.nunique(),
            "example": s.dropna().iloc[0],
        })
    return pd.DataFrame(rows).set_index("column")

print(emp.shape, "| duplicate rows:", emp.duplicated().sum(), "| duplicate IDs:", emp["emp_id"].duplicated().sum())
print(profile(emp))

(302, 13) | duplicate rows: 2 | duplicate IDs: 2
                           dtype  missing_%  unique              example
column
emp_id                       str        0.0     300                 E501
name                         str        0.0     300           Karan Nair
gender                       str        0.0       2                    M
age                        int64        0.0      21                   29
department                   str        0.0      10          Engineering
city                         str        0.0       5            Hyderabad
education                    str        0.0       6               B.Tech
experience_years           int64        0.0      19                    3
join_date         datetime64[us]        0.0     275  2025-10-14 00:00:00
salary                   float64        1.7     255             705000.0
rating                     int64        0.0       5                    1
work_mode                    str        0.0       3               Office
left                         str        0.0       2                   No

Read the table column by column and write down anything odd. There are two duplicate rows (so 300 real employees). department has 10 distinct values, but a company this size probably has about six departments, so spellings need cleaning. salary is the only column with gaps (1.7%). left is text, Yes/No, which you will turn into 0/1 for calculating rates. Nothing is fixed yet; at this stage you are collecting a list.

One column at a time

Next, look at each column on its own: the range and skew of numbers, and the share of each category. Rare categories matter because any rate calculated for them is unreliable.

import pandas as pd

emp = pd.read_csv("employees.csv", parse_dates=["join_date"])
emp["department"] = emp["department"].str.strip().str.title().replace({"Hr": "HR"})

nums = emp[["age", "experience_years", "salary", "rating"]]
summary = nums.describe().T[["min", "50%", "max"]]
summary["skew"] = nums.skew().round(2)
print(summary)

for col in ["department", "education", "work_mode", "left"]:
    shares = emp[col].value_counts(normalize=True).mul(100).round(1)
    print(col, "->", shares.to_dict())

                       min       50%        max  skew
age                   22.0      29.0       44.0  0.75
experience_years       0.0       4.0       21.0  1.19
salary            337000.0  819000.0  1979000.0  0.92
rating                 1.0       3.0        5.0 -0.18
department -> {'Engineering': 32.5, 'Sales': 19.9, 'Support': 16.9, 'Data Science': 12.6, 'Finance': 9.9, 'HR': 8.3}
education -> {'B.Tech': 21.9, 'B.Com': 19.2, 'M.Tech': 17.5, 'B.Sc': 16.9, 'MBA': 16.2, 'MCA': 8.3}
work_mode -> {'Hybrid': 44.7, 'Office': 40.4, 'Remote': 14.9}
left -> {'No': 84.4, 'Yes': 15.6}

After cleaning, there are six departments. Ranges look possible (no age 4 or salary of ₹0). Experience and salary are right-skewed (positive skew): a few senior people with long careers and high pay, so medians describe a typical employee better than means. About 15.6% left, which is the baseline every segment will be compared with. HR is only 8.3% of staff, roughly 25 people, so its attrition rate will be noisy.

Missingness: random or a pattern?

Before you drop or fill the five missing salaries, ask why they are missing. Compare the rows with and without a value. If they look alike, the gaps are probably random and dropping them is fairly safe. If they differ, dropping them biases the result, because you remove a particular kind of person. A missing flag column makes this comparison easy.

import pandas as pd

emp = pd.read_csv("employees.csv", parse_dates=["join_date"])
emp["salary_missing"] = emp["salary"].isna()

print(emp.loc[emp["salary_missing"], ["emp_id", "department", "experience_years", "left"]])
print(emp.groupby("salary_missing")[["experience_years", "rating"]].mean().round(2))
print(emp.groupby("salary_missing")["left"].apply(lambda s: (s == "Yes").mean()).round(2))

    emp_id   department  experience_years left
61    E561           HR                 5   No
71    E571  Engineering                 8  Yes
118   E618  Engineering                 8   No
204   E704  Engineering                 5   No
229   E729        Sales                 2   No
                experience_years  rating
salary_missing
False                        5.1    3.32
True                         5.6    2.80
salary_missing
False    0.15
True     0.20
Name: left, dtype: float64

The five employees without a salary have similar experience and only a slightly lower rating, spread over three departments. With just five rows, these small differences are well within chance, so treating the gaps as random is reasonable here. Record the decision: "5 salaries missing, no clear pattern, excluded from salary analysis only". Statisticians call the three cases missing completely at random, missing at random (explained by other columns) and missing not at random (explained by the missing value itself, such as high earners refusing to state pay). The last one is the dangerous case, because no comparison with other columns can reveal it; only knowing how the data was collected can.

Segments, with sizes

Now the heart of the analysis: the attrition rate for each value of each important category. Always print the group size next to every rate, and draw the overall rate as a reference line, so nobody gets excited about 40% of ten people.

import pandas as pd
import matplotlib.pyplot as plt

emp = pd.read_csv("employees.csv", parse_dates=["join_date"]).drop_duplicates()
emp["department"] = emp["department"].str.strip().str.title().replace({"Hr": "HR"})
emp["left_flag"] = (emp["left"] == "Yes").astype(int)

overall = emp["left_flag"].mean()
print(f"Overall attrition: {overall:.1%} of {len(emp)} employees")

fig, axes = plt.subplots(1, 3, figsize=(11, 3.5), sharey=True)
for ax, col in zip(axes, ["work_mode", "rating", "department"]):
    rate = emp.groupby(col)["left_flag"].agg(["mean", "size"])
    print(col, "|", ", ".join(f"{k}: {m:.0%} of {n:.0f}" for k, (m, n) in rate.iterrows()))
    ax.bar(rate.index.astype(str), rate["mean"] * 100, color="#c2410c")
    ax.axhline(overall * 100, color="gray", linestyle="--", linewidth=1)
    ax.set_title(f"Attrition % by {col}")
    ax.tick_params(axis="x", rotation=45)
plt.tight_layout()
plt.show()

Overall attrition: 15.7% of 300 employees
work_mode | Hybrid: 16% of 134, Office: 16% of 122, Remote: 14% of 44
rating | 1: 40% of 10, 2: 17% of 36, 3: 20% of 133, 4: 11% of 92, 5: 0% of 29
department | Data Science: 13% of 38, Engineering: 16% of 97, Finance: 28% of 29, HR: 4% of 25, Sales: 17% of 60, Support: 14% of 51

The three bar charts share one y-axis, with the dashed line at 15.7%. Work mode makes almost no difference. Rating shows a clear slope: low-rated employees leave more often and nobody rated 5 left, although the rating-1 group is only 10 people. Finance stands out at 28%, but 28% of 29 is just 8 people, so it is a lead to investigate, not a conclusion. To test whether such a difference is larger than chance, use a chi-square test from Hypothesis Testing Basics.

Time and relationships

Finally, check whether the picture changes over time, and how numeric columns relate to the outcome. Group small, old periods together so each group has enough people.

import pandas as pd

emp = pd.read_csv("employees.csv", parse_dates=["join_date"]).drop_duplicates()
emp["left_flag"] = (emp["left"] == "Yes").astype(int)
emp["joined"] = pd.cut(emp["join_date"].dt.year, bins=[1900, 2019, 2021, 2023, 2025],
                       labels=["2019 or before", "2020-21", "2022-23", "2024-25"])

by_period = emp.groupby("joined", observed=True).agg(people=("emp_id", "size"), attrition=("left_flag", "mean"))
print(by_period.round(2))
cols = ["age", "experience_years", "salary", "rating", "left_flag"]
print(emp[cols].corr(method="spearman")["left_flag"].drop("left_flag").round(2))

                people  attrition
joined
2019 or before      34       0.09
2020-21             51       0.12
2022-23             94       0.12
2024-25            121       0.22
age                -0.02
experience_years   -0.01
salary             -0.08
rating             -0.18
Name: left_flag, dtype: float64

People who joined in 2024-25 leave about twice as often as earlier joiners: early attrition is common in many companies, and it suggests looking at hiring and onboarding. Among the numeric columns, only rating has a noticeable (negative) relationship with leaving; age, experience and salary barely matter in this data. Spearman correlation is used because left_flag is 0/1 and salary is skewed. Remember correlation is not causation: a low rating might cause leaving, or an unhappy employee might get a low rating before resigning.

The findings log

End every EDA session by writing a short log. It is the real output of EDA, and it is what your manager reads.

Data: employees.csv, 302 rows, 300 employees after removing 2 duplicate rows.
Fixed: department spellings (10 -> 6 values).
Open: 5 salaries missing, no pattern found; excluded from salary analysis only.
Finding 1: attrition 15.7% overall; 22% among 2024-25 joiners vs 9-12% earlier.
Finding 2: attrition falls as rating rises (40% at rating 1, 0% at rating 5).
Lead: Finance 28% (8 of 29 people) - check with a test and with HR before reporting.
Not a factor: work mode (14-16% in every mode).
Next: chi-square test for department; ask HR how ratings are given and when.

Common mistakes

• Skipping the question. Without one, EDA becomes endless charting.
• Rates without group sizes. A striking percentage from ten people is usually noise.
• Cleaning while exploring, silently. Note every fix in the log so the final numbers can be explained.
• Dropping missing rows without checking the pattern. Compare rows with and without values first.
• Reporting leads as findings. Separate "confirmed" from "worth checking".

Key points

• Start with a one-sentence question, the unit of analysis and the key measure.
• A reusable profile() shows dtype, missing share, distinct values and an example for every column.
• Check skew and rare categories one column at a time.
• Compare rows with and without missing values before deciding how to handle gaps.
• Break the key measure down by segment and period, always with group sizes and the overall baseline.
• Finish with a written log of fixes, findings, leads and next steps.

import pandas as pd

st = pd.read_csv("students.csv", na_values=["AB"]).drop_duplicates()
school = st["school_type"].str.strip().str.lower().str.rstrip(".")
st["school_type"] = school.map({"government": "Government", "govt": "Government", "private": "Private"})

sat = st.dropna(subset=["maths"]).copy()
sat["fail"] = (sat["maths"] < 33).astype(int)
print(len(st), "students,", len(sat), "sat maths")
print(f"Overall fail rate: {sat['fail'].mean():.1%}")
for col in ["school_type", "tuition"]:
    rate = sat.groupby(col)["fail"].agg(["mean", "size"])
    print(col, "|", ", ".join(f"{k}: {m:.1%} of {n:.0f}" for k, (m, n) in rate.iterrows()))

480 students, 478 sat maths
Overall fail rate: 7.9%
school_type | Government: 9.2% of 240, Private: 6.7% of 238
tuition | No: 9.1% of 274, Yes: 6.4% of 204