Data Ethics, Privacy & the DPDP Act

Most data that analysts work with is about people: customers, students, patients, employees. Every row is someone's purchases, marks or salary. Handling it carelessly can embarrass, harm or endanger real people, and in India it can also break the law. Data ethics is the habit of asking, at every step, "should we do this with this data, and how do we protect the people in it?"

This lesson is practical. You will classify the columns of a dataset, keep only what a question needs, mask and pseudonymise identifiers, test whether "anonymous" data really is anonymous, and suppress small numbers in published tables. It ends with the basics of the Digital Personal Data Protection Act, 2023 (DPDP Act). For bias and fairness in models, see Responsible AI; for breach reporting, see Ethics & Law.

Know what you hold

Start by labelling every column. Direct identifiers point to one person on their own: name, phone number, email, Aadhaar or PAN number. Quasi-identifiers do not identify anyone alone, but a combination of them can: age, gender, PIN code, date of birth, job title. Sensitive attributes deserve extra care whatever the law says: health, finances, caste, religion, children's data. Everything else is a measure you analyse.

import numpy as np
import pandas as pd

# Invented practice data: 200 loyalty-card customers (not real people)
rng = np.random.default_rng(11)
n = 200
first = ["Aarav", "Priya", "Kabir", "Meera", "Rohan", "Ananya", "Farhan", "Isha"]
last = ["Sharma", "Nair", "Khan", "Iyer", "Verma", "Gupta"]
pins = {"Kochi": "6820", "Jaipur": "3020", "Pune": "4110", "Lucknow": "2260"}
city = rng.choice(list(pins), size=n, p=[0.4, 0.3, 0.2, 0.1])
customers = pd.DataFrame({
    "name": [f"{rng.choice(first)} {rng.choice(last)}" for _ in range(n)],
    "phone": [f"9{rng.integers(100000000, 999999999)}" for _ in range(n)],
    "email": [f"user{i}@example.com" for i in range(n)],
    "age": rng.integers(18, 70, size=n),
    "gender": rng.choice(["F", "M"], size=n),
    "city": city,
    "pincode": [pins[c] + f"{rng.integers(10, 60)}" for c in city],
    "monthly_spend": rng.gamma(2.0, 1500, size=n).round(-1),
})
print(customers.head(3)[["name", "phone", "age", "city", "pincode", "monthly_spend"]].to_string(index=False))

roles = {
    "direct identifier": ["name", "phone", "email"],
    "quasi-identifier": ["age", "gender", "city", "pincode"],
    "measure": ["monthly_spend"],
}
for role, cols in roles.items():
    print(f"{role:18}", cols)

# Question: how does spending differ by city and age band? Keep only what that needs.
needed = customers[["city", "age", "monthly_spend"]]
print(needed.shape)

       name      phone  age   city pincode  monthly_spend
Farhan Khan 9771848391   44  Kochi  682013         4010.0
Isha Sharma 9618737848   62 Jaipur  302016         3070.0
Ananya Iyer 9713682300   60 Jaipur  302043          710.0
direct identifier  ['name', 'phone', 'email']
quasi-identifier   ['age', 'gender', 'city', 'pincode']
measure            ['monthly_spend']
(200, 3)

The last two lines show data minimisation, the simplest and strongest protection: the question "how does spending differ by city and age?" needs three columns, so the analysis copy has three columns. Data you do not have cannot leak. Ask for the minimum when you request an extract, and say no to "send me everything, just in case".

Masking and pseudonymisation

Sometimes you need to show an identifier, for example to a support team checking an order. Masking hides most of it: a bank SMS shows only the last digits of your account number, and a masked Aadhaar shows only the last four digits. Sometimes you need to link rows about the same person without knowing who it is. Pseudonymisation replaces the identifier with a code.

import hashlib
import hmac
import pandas as pd

phones = pd.Series(["9876543210", "9123456780", "9988776655"])

def mask_phone(p):
    return p[:2] + "X" * 6 + p[-2:]

def mask_email(e):
    user, domain = e.split("@")
    return user[0] + "***@" + domain

print(phones.map(mask_phone).tolist())
print([mask_email(e) for e in ["priya.nair@example.com", "kabir@example.in"]])

plain = hashlib.sha256("9876543210".encode()).hexdigest()
for guess in range(9876540000, 9876550000):           # try 10,000 numbers
    if hashlib.sha256(str(guess).encode()).hexdigest() == plain:
        print("Plain hash reversed by guessing:", guess)
        break

KEY = b"demo-key-for-practice-only"                   # real key: from a secrets store
def pseudonym(value):
    return hmac.new(KEY, value.encode("utf-8"), hashlib.sha256).hexdigest()[:12]
print(phones.map(pseudonym).tolist())

['98XXXXXX10', '91XXXXXX80', '99XXXXXX55']
['p***@example.com', 'k***@example.in']
Plain hash reversed by guessing: 9876543210
['c73ddc3a4b9f', '1713dc177ede', '52ae79434d82']

The middle part is the important lesson. A plain SHA-256 hash of a phone number looks anonymous, but there are only a few billion possible Indian mobile numbers, and a laptop can hash all of them. Here, trying just 10,000 nearby numbers found the original. A keyed hash (HMAC) mixes in a secret key, so nobody without the key can recompute the codes by guessing. The key must be stored apart from the data, for example in a secrets manager, never in the notebook. Even then, pseudonymised data is still personal data: whoever holds the key, or enough other columns, can link it back to people.

Is it really anonymous? k-anonymity

Removing names is not enough. A well-known result from privacy research is that a large share of people can be identified from just their ZIP code, gender and date of birth. The test is simple: group by the quasi-identifiers and count. If a combination appears only once, that row belongs to a recognisable individual. A dataset is k-anonymous when every combination appears at least k times.

import numpy as np
import pandas as pd

# Invented practice data: 200 loyalty-card customers (not real people)
rng = np.random.default_rng(11)
n = 200
first = ["Aarav", "Priya", "Kabir", "Meera", "Rohan", "Ananya", "Farhan", "Isha"]
last = ["Sharma", "Nair", "Khan", "Iyer", "Verma", "Gupta"]
pins = {"Kochi": "6820", "Jaipur": "3020", "Pune": "4110", "Lucknow": "2260"}
city = rng.choice(list(pins), size=n, p=[0.4, 0.3, 0.2, 0.1])
customers = pd.DataFrame({
    "name": [f"{rng.choice(first)} {rng.choice(last)}" for _ in range(n)],
    "phone": [f"9{rng.integers(100000000, 999999999)}" for _ in range(n)],
    "email": [f"user{i}@example.com" for i in range(n)],
    "age": rng.integers(18, 70, size=n),
    "gender": rng.choice(["F", "M"], size=n),
    "city": city,
    "pincode": [pins[c] + f"{rng.integers(10, 60)}" for c in city],
    "monthly_spend": rng.gamma(2.0, 1500, size=n).round(-1),
})

qi = ["age", "gender", "pincode"]
k = customers.groupby(qi)["name"].transform("size")
print("Rows unique on age + gender + PIN:", (k == 1).sum(), "of", len(customers))

general = customers.assign(
    age_band=pd.cut(customers["age"], bins=[17, 29, 44, 59, 69], labels=["18-29", "30-44", "45-59", "60-69"]),
    pin3=customers["pincode"].str[:3] + "XXX",
)
qi2 = ["age_band", "gender", "pin3"]
k2 = general.groupby(qi2, observed=True)["name"].transform("size")
print("After generalising, smallest group size k =", k2.min())
print("Rows in groups smaller than 5:", (k2 < 5).sum())
release = general.loc[k2 >= 5, qi2 + ["monthly_spend"]]
print(release.shape)
print(release.head(3).to_string(index=False))

Rows unique on age + gender + PIN: 200 of 200
After generalising, smallest group size k = 1
Rows in groups smaller than 5: 26
(174, 4)
age_band gender   pin3  monthly_spend
   30-44      F 682XXX         4010.0
   60-69      M 302XXX         3070.0
   60-69      F 302XXX          710.0

With exact age, gender and full PIN code, every customer is unique, so the "anonymous" table identifies all 200 people to anyone who knows a neighbour's age. Generalisation makes values coarser: age bands instead of ages, the first three PIN digits instead of six. That leaves only 26 rows in groups smaller than 5, and those rows are suppressed (left out) before sharing. k-anonymity is a useful check, not a guarantee; if everyone in a group has the same sensitive value, the group still reveals it. For high-risk releases, people use stronger methods such as differential privacy.

Small numbers in published tables

The same risk appears in summary tables. "One customer aged 60-69 in Lucknow spent ₹38,000" points to a person. A common rule is to hide any cell based on fewer than a minimum number of people; the threshold (often 5 or 10) is set by your organisation or the data provider.

import numpy as np
import pandas as pd

# Invented practice data: 200 loyalty-card customers (not real people)
rng = np.random.default_rng(11)
n = 200
pins = {"Kochi": "6820", "Jaipur": "3020", "Pune": "4110", "Lucknow": "2260"}
city = rng.choice(list(pins), size=n, p=[0.4, 0.3, 0.2, 0.1])
customers = pd.DataFrame({
    "age": rng.integers(18, 70, size=n),
    "city": city,
})
customers["age_band"] = pd.cut(customers["age"], bins=[17, 29, 44, 59, 69],
                               labels=["18-29", "30-44", "45-59", "60-69"])
table = pd.crosstab(customers["city"], customers["age_band"])
print(table)

safe = table.astype(str).where(table >= 5, "<5")
print(safe)

age_band  18-29  30-44  45-59  60-69
city
Jaipur       13     12     12     18
Kochi        23     24     27     15
Lucknow       4      5      2      4
Pune         10     15     11      5
age_band 18-29 30-44 45-59 60-69
city
Jaipur      13    12    12    18
Kochi       23    24    27    15
Lucknow     <5     5    <5    <5
Pune        10    15    11     5

where(condition, other) keeps values where the condition is true and replaces the rest. Be aware that suppression can be undone if row or column totals are also published: a reader could subtract the visible cells from the total. When totals are shown, suppress a second cell too, or round the totals.

India's DPDP Act in brief

The Digital Personal Data Protection Act, 2023 applies to digital personal data processed in India, and to processing outside India connected with offering goods or services to people in India. It defines personal data as any data about an individual who is identifiable by or in relation to that data, which includes pseudonymised data. Key terms: the Data Principal is the person the data is about, the Data Fiduciary decides why and how data is processed, and a Data Processor processes it on the fiduciary's behalf. What it means for an analyst, in plain words:

• Purpose and consent: process personal data for a lawful purpose, usually with consent given for a specific purpose, or for certain "legitimate uses" listed in the Act. Reusing data for an unrelated purpose needs a fresh basis.
• Erasure: data must be erased when consent is withdrawn or the purpose is no longer served, unless a law requires keeping it. Old extracts on your laptop count.
• Security: the fiduciary must take reasonable security safeguards; breaches must be reported to the Data Protection Board of India and to affected people. Penalties can reach ₹250 crore.
• Children: for anyone under 18, verifiable parental consent is needed, and tracking, behavioural monitoring and targeted advertising aimed at children are not allowed.
• Rights: people can ask for information about their data, correction and erasure, and can raise grievances.
• Research and statistics: the Act allows exemptions for research, archiving or statistical purposes when the data is not used to take decisions about a specific person and prescribed standards are followed. Ask your legal team before relying on it.

The DPDP Rules, 2025 were notified in November 2025 and phase in over 18 months, so most duties apply from May 2027. This is a summary for learning, not legal advice; check the Act and Rules on the MeitY website and with your company's legal or privacy team.

Common mistakes

• "We removed the names, so it is anonymous." Quasi-identifiers can still single people out; check group sizes.
• Plain hashes of phone numbers or emails. They can be reversed by guessing; use a keyed hash with a secret key.
• Collecting everything. Extra columns add risk without adding answers.
• Keeping extracts forever. Delete working copies when the analysis is done.
• Publishing tiny cells. Suppress counts below your organisation's threshold.

Key points

• Classify columns as direct identifiers, quasi-identifiers, sensitive attributes and measures.
• Data minimisation, keeping only the columns a question needs, is the strongest protection.
• Mask identifiers for display; pseudonymise with a keyed hash (HMAC) for linking; both are still personal data.
• Check k-anonymity by counting quasi-identifier combinations; generalise and suppress small groups.
• Hide small cells in published tables, and watch totals that let readers recompute them.
• The DPDP Act 2023 requires a lawful purpose, consent or a legitimate use, security, erasure and respect for people's rights; most duties under the 2025 Rules apply from May 2027.

import numpy as np
import pandas as pd

# Invented practice data: 200 loyalty-card customers (not real people)
rng = np.random.default_rng(11)
n = 200
first = ["Aarav", "Priya", "Kabir", "Meera", "Rohan", "Ananya", "Farhan", "Isha"]
last = ["Sharma", "Nair", "Khan", "Iyer", "Verma", "Gupta"]
pins = {"Kochi": "6820", "Jaipur": "3020", "Pune": "4110", "Lucknow": "2260"}
city = rng.choice(list(pins), size=n, p=[0.4, 0.3, 0.2, 0.1])
customers = pd.DataFrame({
    "name": [f"{rng.choice(first)} {rng.choice(last)}" for _ in range(n)],
    "phone": [f"9{rng.integers(100000000, 999999999)}" for _ in range(n)],
    "email": [f"user{i}@example.com" for i in range(n)],
    "age": rng.integers(18, 70, size=n),
    "gender": rng.choice(["F", "M"], size=n),
    "city": city,
    "pincode": [pins[c] + f"{rng.integers(10, 60)}" for c in city],
    "monthly_spend": rng.gamma(2.0, 1500, size=n).round(-1),
})

stats = customers.groupby(["city", "gender"])["monthly_spend"].agg(["mean", "size"])
table = stats["mean"].round(-1).astype(int).astype(str).where(stats["size"] >= 5, "<5 customers")
print(table.unstack())

gender      F             M
city
Jaipur   4100          3550
Kochi    2750          2970
Lucknow  2790  <5 customers
Pune     2580          2820