Object Storage, File Systems & Search

By the end of this lesson you will be able to choose between block, file and object storage, design direct uploads with presigned URLs, compare replication with erasure coding, and explain how a search engine finds matching documents in milliseconds with an inverted index.

Three kinds of storage

Databases are good at small, structured records. Photos, videos, PDFs, backups and logs are a different shape: large blobs of bytes. A blob (binary large object) is just a file's raw content. There are three ways to store such data.

Think of a railway parcel office. Block storage is a locker you rent and arrange however you like. File storage is a shared cupboard with labelled shelves and folders. Object storage is the parcel counter itself: you hand over a parcel, get a receipt number (the key), and later show the number to collect it. You never see the shelves, and the office can grow without limit.

Object storage is the default for user-generated files in modern systems. Amazon S3 is the best-known service, and most clouds and open-source tools such as MinIO and Ceph offer S3-compatible APIs. Objects are usually written whole and not edited in place; to change one you upload a new version. In return you get near-unlimited capacity and high durability. AWS, for example, states that S3 is designed for 99.999999999% (eleven nines) durability, and since December 2020 S3 gives strong read-after-write consistency for new and overwritten objects.

The rule for system design: keep the file in object storage, and keep only its key and metadata in the database.

Direct uploads with presigned URLs

If every photo upload passes through your app servers, they spend their time copying bytes. A better pattern is the presigned URL: your server creates a short-lived link that lets the client upload one specific object directly to storage.

1. The app asks your API: "I want to upload a profile photo."
2. Your API checks the user, chooses a key such as avatars/77/2026-11-02.jpg, and signs a URL that expires in a few minutes.
3. The app uploads straight to object storage with that URL.
4. Storage checks the signature and expiry, then stores the file.
5. The app (or a storage event) tells your API the upload is complete, and the API saves the key in the database.

The signature is a keyed hash (an HMAC) of the request details, made with a secret that only your server and the storage service know. Here is a simplified version of the idea:

import hashlib, hmac

SECRET = b"demo-secret-never-hardcode-real-keys"

def sign(method, key, expires_at):
    message = f"{method}\n{key}\n{expires_at}".encode()
    return hmac.new(SECRET, message, hashlib.sha256).hexdigest()[:16]

def presign(method, key, now, valid_for=300):
    expires = now + valid_for
    return {"method": method, "key": key, "expires": expires, "sig": sign(method, key, expires)}

def storage_accepts(req, now):
    if now > req["expires"]:
        return "rejected: link expired"
    good = sign(req["method"], req["key"], req["expires"])
    if not hmac.compare_digest(good, req["sig"]):
        return "rejected: bad signature"
    return f"accepted {req['method']} {req['key']}"

url = presign("PUT", "avatars/77/2026-11-02.jpg", now=1_000)
print(storage_accepts(url, now=1_100))
print(storage_accepts(url, now=1_400))
tampered = dict(url, key="avatars/78/2026-11-02.jpg")     # try to overwrite someone else's photo
print(storage_accepts(tampered, now=1_100))

accepted PUT avatars/77/2026-11-02.jpg
rejected: link expired
rejected: bad signature

Changing the key breaks the signature, so a user cannot reuse their link to overwrite another user's file. Real services use a more detailed signing format, but the idea is the same. Large files use multipart upload: the file is sent in parts that can be retried individually, then joined.

Keeping data safe: replication vs erasure coding

Storage systems lose disks every day, so they store data redundantly. Two approaches:

• Replication: keep 3 full copies. Survives 2 lost copies. Uses 3 times the space.
• Erasure coding: split data into k data pieces and add m parity pieces, calculated so that any k of the k + m pieces can rebuild the data. A 6 + 3 scheme survives 3 lost pieces using only 1.5 times the space.

def scheme(name, data_parts, extra_parts):
    total = data_parts + extra_parts
    return f"{name:22} survives {extra_parts} lost pieces, stores {total / data_parts:.2f}x the data"

print(scheme("3 full copies", 1, 2))
print(scheme("Erasure code 6+3", 6, 3))
print(scheme("Erasure code 10+4", 10, 4))

petabytes = 2
print(f"\nFor {petabytes} PB of photos:")
for name, factor in [("3 copies", 3.0), ("6+3", 1.5), ("10+4", 1.4)]:
    print(f"  {name:8} needs {petabytes * factor:.1f} PB of raw disk")

3 full copies          survives 2 lost pieces, stores 3.00x the data
Erasure code 6+3       survives 3 lost pieces, stores 1.50x the data
Erasure code 10+4      survives 4 lost pieces, stores 1.40x the data

For 2 PB of photos:
  3 copies needs 6.0 PB of raw disk
  6+3      needs 3.0 PB of raw disk
  10+4     needs 2.8 PB of raw disk

Erasure coding saves a lot of disk, but reading or rebuilding needs pieces from several machines and more CPU. Systems often replicate fresh, frequently read data and erasure-code older, colder data. This is also why object storage offers cheaper "infrequent access" and "archive" tiers, with lifecycle rules that move old objects there automatically.

Distributed file systems

Before object storage became common, big data systems used distributed file systems. Google described the Google File System (GFS) in a 2003 paper: large files are split into big chunks, each chunk is replicated on several machines, and a master server keeps track of where chunks live. Hadoop's HDFS follows the same design. These suit huge files read in long sequential scans, such as logs for batch analytics. For user uploads in web apps, object storage is usually the simpler choice.

Search systems: the inverted index

A database index on name helps exact lookups and prefixes. It cannot efficiently answer "find products whose description mentions cotton and kurta". Text search needs an inverted index: a map from each word to the list of documents that contain it, like the index at the back of a book that lists the pages for each term.

Building it has three steps:

1. Tokenise: split text into words (tokens) and lowercase them.
2. Normalise: remove very common words and reduce words to a base form if needed.
3. Index: for each token, record which documents contain it and how often.

To search, look up each query word, intersect the document lists and rank the results. Ranking formulas such as TF-IDF and BM25 give more weight to words that appear often in a document but rarely across all documents.

import math, re
from collections import defaultdict

docs = {
    1: "Cotton kurta for men, handloom cotton from Lucknow",
    2: "Silk saree with zari border, Kanchipuram silk",
    3: "Cotton saree, light and breathable for summer",
    4: "Kurta pyjama set in linen, festive wear",
}
STOP = {"for", "with", "and", "in", "from", "the", "set"}

def tokens(text):
    return [w for w in re.findall(r"[a-z]+", text.lower()) if w not in STOP]

index = defaultdict(dict)                     # word -> {doc_id: count}
for doc_id, text in docs.items():
    for w in tokens(text):
        index[w][doc_id] = index[w].get(doc_id, 0) + 1

def search(query):
    words = tokens(query)
    matching = set.intersection(*(set(index[w]) for w in words)) if words else set()
    def score(d):                             # simple TF-IDF
        return sum(index[w][d] * math.log(len(docs) / len(index[w])) for w in words)
    return sorted(((round(score(d), 2), d) for d in matching), reverse=True)

print("cotton ->", dict(index["cotton"]))
print("search 'cotton kurta':", search("cotton kurta"))
print("search 'saree':", search("saree"))

cotton -> {1: 2, 3: 1}
search 'cotton kurta': [(2.08, 1)]
search 'saree': [(0.69, 3), (0.69, 2)]

Real search engines such as Elasticsearch and OpenSearch (both built on Apache Lucene) add typo tolerance, synonyms, Indian-language analysers, filters, facets and distributed shards. In a system design, the search engine is a secondary store: the database remains the source of truth, and changes flow to the search index through a queue or a change-data-capture stream, usually becoming searchable within a second or so. You will see this flow in Message Queues & Event Streaming.

Common mistakes

• Storing images or PDFs inside database rows, making backups and replicas huge and slow.
• Uploading large files through app servers instead of presigned direct uploads.
• Public buckets by accident, exposing private documents. Keep buckets private and serve through signed URLs or a CDN.
• Using LIKE '%word%' for search on large tables, which scans every row.
• Treating the search index as the source of truth. It can be rebuilt; the database cannot.

Key points

• Block storage is a raw disk, file storage is shared folders, object storage is keys and blobs over HTTP.
• Store files in object storage and only keys and metadata in the database.
• Presigned URLs let clients upload directly and safely, with an expiry and a signature.
• Erasure coding survives as many failures as replication with far less disk, at some CPU and read cost.
• Search engines use inverted indexes and ranking such as TF-IDF or BM25, fed from the main database.

import math, re
from collections import defaultdict

docs = {
    1: "Cotton kurta for men, handloom cotton from Lucknow",
    2: "Silk saree with zari border, Kanchipuram silk",
    3: "Cotton saree, light and breathable for summer",
    4: "Kurta pyjama set in linen, festive wear",
}
STOP = {"for", "with", "and", "in", "from", "the", "set"}

def tokens(text):
    return [w for w in re.findall(r"[a-z]+", text.lower()) if w not in STOP]

index = defaultdict(dict)
for doc_id, text in docs.items():
    for w in tokens(text):
        index[w][doc_id] = index[w].get(doc_id, 0) + 1

def search_any(query):
    words = [w for w in tokens(query) if w in index]
    matching = set().union(*(set(index[w]) for w in words))
    def score(d):
        return sum(index[w].get(d, 0) * math.log(len(docs) / len(index[w])) for w in words)
    return sorted(((round(score(d), 2), d) for d in matching), reverse=True)

print(search_any("linen cotton"))

[(1.39, 4), (1.39, 1), (0.69, 3)]