Data Sources & Data Collection

By the end of this lesson you will know the main places analysts get data from, the difference between structured and unstructured data, how to read JSON from an API into a table, and how to design a data collection form that does not create a cleaning nightmare later.

Think of a dabbawala in Mumbai. Each tiffin box carries a painted code that says where it came from and where it must go. Without the code the system collapses. Data is the same: you must always know where it came from, how it was collected and what each field means.

Where data comes from

Internal data is created inside your organisation. External data comes from outside it. Most analysis starts with internal data.

Primary data is collected by you for your question, such as a customer survey. Secondary data was collected by someone else for another purpose, such as an orders table built for billing. Most analyst work uses secondary data, so you must check that it fits your question.

Structured, semi-structured and unstructured

Structured data sits in rows and columns with fixed types, like a database table or a clean CSV. Semi-structured data has labels but no fixed table shape, like JSON from an API. Unstructured data has no labels at all: emails, call recordings, photos, free-text reviews.

Analysts mostly work with structured data and turn semi-structured data into tables. An API (Application Programming Interface) is a way for one program to ask another for data, and many APIs reply in JSON. Here is a small JSON reply from an imaginary delivery partner, turned into a table.

import json
import pandas as pd

reply = """
[
  {"awb": "DL1001", "city": "Jaipur", "status": "delivered", "charges": {"base": 40, "cod_fee": 0}},
  {"awb": "DL1002", "city": "Kochi", "status": "in_transit", "charges": {"base": 55, "cod_fee": 20}},
  {"awb": "DL1003", "city": "Pune", "status": "delivered", "charges": {"base": 45, "cod_fee": 20}}
]
"""

rows = json.loads(reply)
table = pd.json_normalize(rows)
table["total_charge"] = table["charges.base"] + table["charges.cod_fee"]
print(table)

      awb    city      status  charges.base  charges.cod_fee  total_charge
0  DL1001  Jaipur   delivered            40                0            40
1  DL1002   Kochi  in_transit            55               20            75
2  DL1003    Pune   delivered            45               20            65

What you should see: a neat table. json_normalize flattened the nested charges object into two columns with dotted names. Once data is a table, every tool in this course can work with it.

Step 1: check that the data is fresh

Before using any source, check how up to date it is. A dashboard built on data that stopped loading three weeks ago is worse than no dashboard. A simple freshness check finds the first and last dates and counts rows per month.

SELECT SUBSTR(order_date, 1, 7) AS month,
       COUNT(*) AS orders,
       MIN(order_date) AS first_order,
       MAX(order_date) AS last_order
FROM orders
GROUP BY SUBSTR(order_date, 1, 7)
ORDER BY month;

month   | orders | first_order | last_order
--------+--------+-------------+-----------
2025-01 | 3      | 2025-01-05  | 2025-01-20
2025-02 | 3      | 2025-02-02  | 2025-02-28
2025-03 | 3      | 2025-03-09  | 2025-03-22
2025-04 | 3      | 2025-04-01  | 2025-04-18
2025-05 | 3      | 2025-05-02  | 2025-05-20
(5 rows)

What you should see: five months with three orders each, and no month missing. If a month were missing, or the last date were weeks old, you would ask the data owner before analysing anything.

Step 2: collect data that is easy to clean

When you design a form or survey, you decide how much cleaning you will do later. Free-text boxes invite chaos. Look at what happens when people type their city by hand.

import pandas as pd

typed = pd.Series(["Bengaluru", "bangalore", "Bangalore ", "BLR", "Bengaluru",
                   "Pune", "pune", "Poona", "Kochi", "Cochin"])
print(typed.value_counts())
print("Distinct spellings:", typed.nunique())

Bengaluru     2
bangalore     1
Bangalore     1
BLR           1
Pune          1
pune          1
Poona         1
Kochi         1
Cochin        1
Name: count, dtype: int64
Distinct spellings: 9

What you should see: ten answers for three real cities, but nine different spellings. A drop-down list would have given three clean values with no extra work. Good collection rules:

1. Use drop-downs, radio buttons and date pickers instead of free text where you can.
2. Make important fields required, and validate formats (a 10-digit mobile number, a 6-digit PIN code).
3. Ask only what you need. Every extra question lowers the number of people who finish the form, and collecting personal data you do not need is a privacy risk.
4. Record when and how each answer was collected (date, channel, form version).

Bias in collection

How data is collected can make it unrepresentative. Sampling bias happens when the people in your data are not like the people you want to understand. A feedback form shown only after a successful delivery will never hear from customers whose orders failed. An online survey misses people who rarely use the internet.

Ask of every source: who is missing from this data, and why? The Statistics course covers sampling methods in depth.

Step 3: write a data dictionary

A data dictionary is a short document that explains each column: its name, type, meaning and an example. It is the painted code on the tiffin box. You can start one automatically and then add meanings by hand.

import pandas as pd

sales = pd.read_csv("sales.csv")
dictionary = pd.DataFrame({
    "type": sales.dtypes.astype(str),
    "distinct": sales.nunique(),
    "blanks": sales.isna().sum(),
    "example": sales.iloc[0],
})
print(dictionary)

                 type  distinct  blanks      example
order_id          str      1209       0     ORD10001
order_date        str       347       0   2025-01-01
city              str        20       0       Mumbai
state             str         9       0  Maharashtra
category          str         6       0       Sports
product           str        29       0     Yoga Mat
quantity      float64         8       6          1.0
unit_price      int64        25       0          699
discount        int64         5       0            0
payment_mode      str        12      11          UPI

What you should see: one row per column with its type, number of distinct values, blanks and an example. Now add a "meaning" column yourself: for example, discount is a percentage, not rupees. That single note prevents a costly mistake.

Common mistakes

• Using a source without knowing how it was collected. A "customers" table might include test accounts and staff.
• Not checking freshness. Always check the latest date before analysing.
• Free text everywhere. Design forms with fixed choices so the data arrives clean.
• Ignoring who is missing. Biased collection gives confident but wrong answers.

Key points

• Internal sources (databases, CRM, ERP, analytics tools) are the analyst's main data; external sources add context.
• Structured data is in tables; semi-structured data such as JSON can be flattened into tables.
• Check freshness and completeness before using any source.
• Good forms use fixed choices, validation and only the questions you need.
• Ask who is missing from the data, and write a data dictionary for every dataset you use.

import pandas as pd

emp = pd.read_csv("employees.csv")
dictionary = pd.DataFrame({
    "type": emp.dtypes.astype(str),
    "distinct": emp.nunique(),
    "blanks": emp.isna().sum(),
    "example": emp.iloc[0],
})
print(dictionary)
# left: "Yes" if the employee has left the company, "No" if still employed.
# Question for HR: does "Yes" include people who retired or were transferred,
# or only people who resigned?