TF-IDF

By the end of this lesson you will understand how TF-IDF decides which words matter in a document. You will calculate it by hand, check your numbers against scikit-learn, pull out keywords, and build a small search engine for college FAQs.

Raw word counts treat every word as equally important. But in a set of railway complaints, the word "train" appears in nearly every message, so it tells you almost nothing. A word like "AC" or "refund" appears in only a few messages, and those messages are about something specific. TF-IDF gives rare, specific words more weight.

Think of your class WhatsApp group. If everyone says "ok", that word tells you nothing about who wrote a message. If only Farhan ever writes "bro", seeing "bro" points straight at Farhan. Rare words are better clues.

The two parts

TF stands for term frequency: how often a word appears in this document. A word repeated three times in a complaint is probably important to it.

IDF stands for inverse document frequency: how rare the word is across all documents. If a word appears in many documents, its IDF is low. If it appears in only a few, its IDF is high.

The TF-IDF weight is simply TF multiplied by IDF. scikit-learn calculates IDF with this formula (the default, called "smooth" IDF):

idf(word) = ln((1 + n) / (1 + df)) + 1

n  = number of documents
df = number of documents that contain the word
ln = natural logarithm

The + 1 parts stop division by zero and make sure no word gets a weight of exactly zero. Finally, each document's row is scaled so its length is 1. This normalisation means long documents do not get bigger scores just for being long.

Calculating TF-IDF by hand

Step 1. Count document frequency for each word.

Step 2. Compute IDF with the formula above.

Step 3. Multiply each word's count by its IDF, then divide the row by its length.

Step 4. Compare with TfidfVectorizer.

import math
from sklearn.feature_extraction.text import TfidfVectorizer

docs = ["train late again", "train ac not working", "refund for cancelled train"]
n = len(docs)
vocab = sorted({w for d in docs for w in d.split()})

df = {w: sum(w in d.split() for d in docs) for w in vocab}                 # Step 1
idf = {w: math.log((1 + n) / (1 + df[w])) + 1 for w in vocab}               # Step 2

doc = docs[1].split()
raw = {w: doc.count(w) * idf[w] for w in vocab if w in doc}                 # Step 3
length = math.sqrt(sum(v * v for v in raw.values()))
by_hand = {w: round(v / length, 3) for w, v in raw.items()}
print("By hand:     ", by_hand)

vec = TfidfVectorizer().fit(docs)                                           # Step 4
row = vec.transform([docs[1]]).toarray()[0]
lib = {w: round(float(x), 3) for w, x in zip(vec.get_feature_names_out(), row) if x > 0}
print("scikit-learn:", lib)

By hand:      {'ac': 0.546, 'not': 0.546, 'train': 0.323, 'working': 0.546}
scikit-learn: {'ac': 0.546, 'not': 0.546, 'train': 0.323, 'working': 0.546}

The two results match. "train" is in all three documents, so it gets the smallest weight. "ac", "not" and "working" appear only once, so they share the biggest weights. Note that scikit-learn's default tokeniser also accepts the two-letter word "ac".

Finding keywords

Because TF-IDF rewards words that are common in one document but rare elsewhere, the top-weighted words of a document make good keywords. Here we find two keywords for each invented complaint.

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

complaints = [
    "The AC in coach B2 was not working and the coach was very hot",
    "Food from the pantry was cold and the pantry staff was rude",
    "Train reached Jaipur three hours late, the delay was not announced",
    "The toilet in coach S4 was dirty and there was no water in the toilet",
]
vec = TfidfVectorizer(stop_words="english")
X = vec.fit_transform(complaints)
words = vec.get_feature_names_out()

for i, text in enumerate(complaints):
    row = X[i].toarray()[0]
    top = np.argsort(row)[::-1][:2]
    print(f"{', '.join(words[top]):<16} <- {text[:45]}...")

coach, working   <- The AC in coach B2 was not working and the co...
pantry, staff    <- Food from the pantry was cold and the pantry ...
jaipur, train    <- Train reached Jaipur three hours late, the de...
toilet, s4       <- The toilet in coach S4 was dirty and there wa...

Each complaint is summed up by the words that make it different: the pantry complaint by "pantry", the toilet complaint by "toilet", and so on. This is a cheap way to tag documents or suggest search keywords. We used stop_words="english" here because this is a topic task, not sentiment.

A tiny search engine

TF-IDF also powers simple search. Turn every document and the query into TF-IDF vectors, then rank documents by cosine similarity, a score from 0 to 1 that measures how much two vectors point in the same direction. You will study cosine similarity in detail in Word Embeddings & Cosine Similarity.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

faqs = [  # invented college FAQs
    "How do I pay the semester fee online?",
    "What is the last date for the scholarship form?",
    "How can I get a duplicate ID card?",
    "Where do I collect the hostel room key?",
    "Can I pay the hostel fee in two instalments?",
]
vec = TfidfVectorizer(stop_words="english")
faq_vectors = vec.fit_transform(faqs)

for query in ["hostel fee instalment", "lost my id card"]:
    scores = cosine_similarity(vec.transform([query]), faq_vectors)[0]
    best = scores.argmax()
    print(f"{query!r} -> {faqs[best]} (score {scores[best]:.2f})")

'hostel fee instalment' -> Can I pay the hostel fee in two instalments? (score 0.66)
'lost my id card' -> How can I get a duplicate ID card? (score 0.82)

The first query finds the instalment FAQ, but look at why: it matched on "hostel" and "fee". The query word "instalment" did not match "instalments", because they are different tokens. The second finds the ID card question even though the query says "lost", which no FAQ contains. The shared words "id" and "card" were enough. But a query like "fees in parts" would fail, because "parts" never appears and "fees" is not the same token as "fee". Search that understands meaning needs embeddings, covered in Sentence Embeddings & Semantic Search.

Useful TfidfVectorizer settings

• sublinear_tf=True uses 1 + log(count) instead of the raw count, so a word repeated ten times does not dominate.
• norm=None turns off the length-1 scaling, if you ever need raw weights.
• All CountVectorizer options work too: ngram_range, min_df, max_df, stop_words.

To use TF-IDF features in a full classifier with typos and Hinglish, read Text Classification with TF-IDF in the Machine Learning course.

Common mistakes

• Fitting the vectorizer on test documents. IDF must come from training text only.
• Comparing TF-IDF scores from different fitted vectorizers. Each fit has its own vocabulary and IDF values.
• Expecting synonyms to match. TF-IDF only matches exact tokens: "fee" and "fees" are different.

Key points

• TF-IDF = term frequency (how often in this document) × inverse document frequency (how rare overall).
• scikit-learn uses smooth IDF, ln((1 + n) / (1 + df)) + 1, and scales each row to length 1.
• Words in every document get low weights; distinctive words get high weights.
• The top TF-IDF words of a document are good keywords.
• TF-IDF plus cosine similarity gives a simple, fast search engine that matches exact words.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

faqs = ["How do I pay the semester fee online?",
        "What is the last date for the scholarship form?",
        "How can I get a duplicate ID card?",
        "Where do I collect the hostel room key?",
        "Can I pay the hostel fee in two instalments?",
        "What documents are needed for admission?",
        "When does the library close?",
        "How do I change my registered mobile number?"]
vec = TfidfVectorizer(stop_words="english")
faq_vectors = vec.fit_transform(faqs)
for query in ["library timing", "update phone number"]:
    scores = cosine_similarity(vec.transform([query]), faq_vectors)[0]
    best = scores.argmax()
    print(f"{query!r} -> {faqs[best]} (score {scores[best]:.2f})")

'library timing' -> When does the library close? (score 0.58)
'update phone number' -> How do I change my registered mobile number? (score 0.50)