Text Classification with TF-IDF
A large share of business data is text: support tickets, product reviews, chat messages, complaint emails. Routing each support ticket to the right team (delivery, refunds, payments, accounts) by hand is slow and boring work, and a simple text classifier can do most of it in milliseconds.
You saw word counts briefly in Naive Bayes. This lesson builds a proper text classifier the way it is still done in many companies: TF-IDF features, word and character n-grams, a linear model, and a look inside to see which words drive each decision. It works with very little computing power, trains in seconds and is easy to explain, which makes it the right first step before reaching for large language models.
From text to numbers
Models need numbers, so text must be turned into a table. The bag of words approach makes one column per word in the vocabulary and counts how often each word appears in each document. Word order is lost, but the counts carry a lot of meaning.
TF-IDF (term frequency times inverse document frequency) improves on raw counts. A word that appears in almost every ticket, like "order", gets a low weight; a word that appears in few tickets, like "otp", gets a high weight because it says more about the ticket. scikit-learn's TfidfVectorizer also lowercases the text and normalises each row to length 1.
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
tickets = ["my order is not delivered",
"refund for my order not received",
"otp not coming for login"]
counts = CountVectorizer().fit(tickets)
print("Vocabulary:", list(counts.get_feature_names_out()))
print("Counts for ticket 1:", counts.transform(tickets[:1]).toarray()[0])
tfidf = TfidfVectorizer().fit(tickets)
row = tfidf.transform(tickets[2:]).toarray()[0]
for word, weight in zip(tfidf.get_feature_names_out(), row):
if weight > 0:
print(f"{word:<8} {weight:.3f}")Vocabulary: ['coming', 'delivered', 'for', 'is', 'login', 'my', 'not', 'order', 'otp', 'received', 'refund'] Counts for ticket 1: [0 1 0 1 0 1 1 1 0 0 0] coming 0.505 for 0.384 login 0.505 not 0.298 otp 0.505
In the third ticket, "coming", "login" and "otp" each get 0.505 because they appear in only one ticket, while "not", which appears in all three, gets only 0.298. TF-IDF automatically emphasises the words that make a ticket different from the others.
The result of transform is a sparse matrix: it stores only the non-zero values, because each ticket uses a tiny part of the vocabulary. Linear models such as LogisticRegression and LinearSVC work directly with sparse input and handle tens of thousands of columns easily.
A ticket router
The examples below use a generator of 1,200 short support tickets for an online shop, in four classes. Some tickets mention two problems, which makes them genuinely ambiguous, as real tickets are. The pipeline joins the vectorizer and the classifier, so fit learns the vocabulary from training text only. ngram_range=(1, 2) adds bigrams (pairs of neighbouring words), so "not delivered" and "not received" become features of their own.
import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
PROBLEMS = {
"delivery": ["order not delivered yet", "parcel is delayed again", "delivery boy did not come",
"tracking shows shipped but not received", "where is my package", "kab tak aayega order",
"wrong address on delivery", "courier marked delivered but i got nothing"],
"refund": ["refund not received", "return picked up but money not credited", "paisa wapas nahi aaya",
"want to return this product", "refund status still pending", "cancelled order but no refund",
"refund amount is less", "exchange or refund for damaged item"],
"payment": ["money debited but order failed", "upi payment failed", "charged twice for one order",
"payment pending for hours", "card declined at checkout", "amount cut from bank but no order",
"emi option not working", "cash on delivery not available"],
"account": ["cannot login to my account", "otp not coming", "need to change mobile number",
"password reset link not working", "account blocked suddenly", "update my email address",
"delete my account", "profile name is wrong"],
}
def make_tickets(n=1200, seed=42):
rng = np.random.default_rng(seed)
openers = ["", "hi", "hello team", "sir", "please help", "urgent"]
closers = ["", "order id 4521", "please check", "since 3 days", "very bad service", "pls resolve asap"]
labels = rng.choice(list(PROBLEMS), n)
rows = []
for label in labels:
text = f"{rng.choice(openers)} {rng.choice(PROBLEMS[label])}"
if rng.random() < 0.15: # a second, unrelated problem
other = rng.choice([k for k in PROBLEMS if k != label])
text += f" and also {rng.choice(PROBLEMS[other])}"
rows.append(f"{text} {rng.choice(closers)}".strip())
return pd.DataFrame({"text": rows, "team": labels})
df = make_tickets()
print(df.sample(3, random_state=1).to_string(index=False))
X_train, X_test, y_train, y_test = train_test_split(
df["text"], df["team"], test_size=0.25, random_state=42, stratify=df["team"])
router = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LogisticRegression(C=10))
router.fit(X_train, y_train)
print(classification_report(y_test, router.predict(X_test), digits=3))
new = ["paisa abhi tak wapas nahi aaya", "otp nahi aa raha login ke liye"]
print(dict(zip(new, router.predict(new)))) text team
please help order not delivered yet very bad service delivery
hello team otp not coming since 3 days account
please help paisa wapas nahi aaya and also kab tak aayega order since 3 days refund
precision recall f1-score support
account 1.000 1.000 1.000 76
delivery 1.000 0.947 0.973 75
payment 0.987 1.000 0.994 77
refund 0.960 1.000 0.980 72
accuracy 0.987 300
macro avg 0.987 0.987 0.986 300
weighted avg 0.987 0.987 0.987 300
{'paisa abhi tak wapas nahi aaya': 'refund', 'otp nahi aa raha login ke liye': 'account'}The router gets 98.7% of the test tickets right. Generated tickets are cleaner than real ones, so expect lower scores on real data, but the method is exactly the same. It also routes the two new Hinglish tickets correctly, because words such as "paisa", "wapas" and "otp" appeared in training. The few mistakes are tickets with two problems, where even a person might hesitate.
What did the model learn?
A linear text model is easy to inspect. For each class, coef_ holds one weight per vocabulary entry; the largest weights are the words that most strongly push a ticket into that class. Checking them is the fastest way to catch problems, such as an order ID or an agent's name that leaked into the training text.
import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
PROBLEMS = {
"delivery": ["order not delivered yet", "parcel is delayed again", "delivery boy did not come",
"tracking shows shipped but not received", "where is my package", "kab tak aayega order",
"wrong address on delivery", "courier marked delivered but i got nothing"],
"refund": ["refund not received", "return picked up but money not credited", "paisa wapas nahi aaya",
"want to return this product", "refund status still pending", "cancelled order but no refund",
"refund amount is less", "exchange or refund for damaged item"],
"payment": ["money debited but order failed", "upi payment failed", "charged twice for one order",
"payment pending for hours", "card declined at checkout", "amount cut from bank but no order",
"emi option not working", "cash on delivery not available"],
"account": ["cannot login to my account", "otp not coming", "need to change mobile number",
"password reset link not working", "account blocked suddenly", "update my email address",
"delete my account", "profile name is wrong"],
}
def make_tickets(n=1200, seed=42):
rng = np.random.default_rng(seed)
openers = ["", "hi", "hello team", "sir", "please help", "urgent"]
closers = ["", "order id 4521", "please check", "since 3 days", "very bad service", "pls resolve asap"]
labels = rng.choice(list(PROBLEMS), n)
rows = []
for label in labels:
text = f"{rng.choice(openers)} {rng.choice(PROBLEMS[label])}"
if rng.random() < 0.15:
other = rng.choice([k for k in PROBLEMS if k != label])
text += f" and also {rng.choice(PROBLEMS[other])}"
rows.append(f"{text} {rng.choice(closers)}".strip())
return pd.DataFrame({"text": rows, "team": labels})
df = make_tickets()
router = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LogisticRegression(C=10)).fit(df["text"], df["team"])
vec, clf = router[0], router[-1]
words = vec.get_feature_names_out()
print("Vocabulary size:", len(words))
for team, weights in zip(clf.classes_, clf.coef_):
top = np.argsort(weights)[::-1][:5]
print(f"{team:<9}", ", ".join(words[top]))Vocabulary size: 569 account account, account and, my, otp, not coming delivery delivered, address on, wrong address, again and, delivery payment failed and, failed, payment, order and, option not refund refund, refund not, return, refund and, product and
Most top words make sense: "refund", "return", "otp", "payment", "failed". But look at the bigrams ending in "and", such as "account and" and "failed and". They come from tickets with two problems, where the first problem decides the team. The model has learned how our labels were assigned, not just what the problem is. On real data, a discovery like this starts a useful conversation: should a two-problem ticket go to the first team, or be split in two?
Typos and Hinglish: character n-grams
Real customers type fast on phones: "refnd", "delivry", "pymnt", and they mix Hindi and English in Roman script. A word-level vocabulary treats "refnd" as an unknown word and ignores it. Character n-grams split text into short overlapping pieces such as "ref", "efn" and "fnd", so a misspelt word still shares most of its pieces with the correct one. Use analyzer="char_wb", which builds character n-grams only inside word boundaries.
import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline, make_union
PROBLEMS = {
"delivery": ["order not delivered yet", "parcel is delayed again", "delivery boy did not come",
"tracking shows shipped but not received", "where is my package", "kab tak aayega order",
"wrong address on delivery", "courier marked delivered but i got nothing"],
"refund": ["refund not received", "return picked up but money not credited", "paisa wapas nahi aaya",
"want to return this product", "refund status still pending", "cancelled order but no refund",
"refund amount is less", "exchange or refund for damaged item"],
"payment": ["money debited but order failed", "upi payment failed", "charged twice for one order",
"payment pending for hours", "card declined at checkout", "amount cut from bank but no order",
"emi option not working", "cash on delivery not available"],
"account": ["cannot login to my account", "otp not coming", "need to change mobile number",
"password reset link not working", "account blocked suddenly", "update my email address",
"delete my account", "profile name is wrong"],
}
def make_tickets(n=1200, seed=42):
rng = np.random.default_rng(seed)
openers = ["", "hi", "hello team", "sir", "please help", "urgent"]
closers = ["", "order id 4521", "please check", "since 3 days", "very bad service", "pls resolve asap"]
labels = rng.choice(list(PROBLEMS), n)
rows = []
for label in labels:
text = f"{rng.choice(openers)} {rng.choice(PROBLEMS[label])}"
if rng.random() < 0.15:
other = rng.choice([k for k in PROBLEMS if k != label])
text += f" and also {rng.choice(PROBLEMS[other])}"
rows.append(f"{text} {rng.choice(closers)}".strip())
return pd.DataFrame({"text": rows, "team": labels})
def add_typos(text, rng, rate=0.7):
"""Drop one inner letter from some longer words, like hurried phone typing."""
words = []
for w in text.split():
if len(w) > 4 and rng.random() < rate:
i = rng.integers(1, len(w) - 1)
w = w[:i] + w[i + 1:]
words.append(w)
return " ".join(words)
df = make_tickets()
X_train, X_test, y_train, y_test = train_test_split(
df["text"], df["team"], test_size=0.25, random_state=42, stratify=df["team"])
rng = np.random.default_rng(7)
X_test_typos = X_test.apply(lambda t: add_typos(t, rng))
print("Example with typos:", X_test_typos.iloc[0])
models = {
"word 1-2 grams": TfidfVectorizer(ngram_range=(1, 2)),
"char_wb 2-4 grams": TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4)),
"word + char": make_union(TfidfVectorizer(ngram_range=(1, 2)),
TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4))),
}
for name, vectorizer in models.items():
model = make_pipeline(vectorizer, LogisticRegression(C=10)).fit(X_train, y_train)
print(f"{name:<18} clean test {model.score(X_test, y_test):.3f} "
f"test with typos {model.score(X_test_typos, y_test):.3f}")Example with typos: plese help payment pendig for hurs ordr id 4521 word 1-2 grams clean test 0.987 test with typos 0.910 char_wb 2-4 grams clean test 0.937 test with typos 0.927 word + char clean test 0.983 test with typos 0.947
Heavy typos cut the word model's accuracy from 0.987 to 0.910. The character model barely notices them (0.937 to 0.927), but it is weaker on clean text. Combining both with make_union, which places the two feature sets side by side, keeps almost all of the clean accuracy and is the best of the three on typos (0.947). Word plus character features is a common choice for messy, informal text.
When to go beyond TF-IDF
TF-IDF with a linear model is a strong, cheap baseline, and many production systems still use it. Its main limit is that it only knows the words it has seen: "my parcel never showed up" shares no words with "order not delivered", so the model cannot tell they mean the same thing. Options that understand meaning, in rough order of cost:
- Sentence embeddings plus logistic regression: a pretrained model turns each ticket into a vector of meaning (see Embeddings & Semantic Search); you train only the small classifier on top.
- Fine-tuning a transformer model on your labelled tickets (see Deep Learning: CNNs, RNNs & Transformers).
- Asking an LLM to classify each ticket with a prompt, useful when you have almost no labels, but slower and costlier per ticket.
Whichever you choose, keep the TF-IDF model as the baseline to beat.
Common mistakes
- Fitting the vectorizer on all the text before splitting. The vocabulary and IDF weights then include test documents; keep the vectorizer inside the pipeline.
- Passing a list of columns to a vectorizer in a
ColumnTransformer. Text vectorizers expect a single column name as a string, such as"text", not["text"]. - Removing words like "not" as stop words. In tickets and reviews, "not received" and "received" mean opposite things.
- Ignoring typos and mixed languages. Try
analyzer="char_wb"when users type informally. - Never reading the top features. Leaked IDs and names show up there first.
Key points
- Bag of words turns text into word counts; TF-IDF down-weights words that appear everywhere.
- Vectorizers produce sparse matrices that linear models handle very efficiently.
ngram_range=(1, 2)adds word pairs such as "not delivered", which capture short phrases.- Character n-grams (
analyzer="char_wb") make models robust to typos and informal spelling. - Inspect
coef_withget_feature_names_out()to see which words drive each class. - Keep TF-IDF plus a linear model as the baseline before trying embeddings, transformers or LLMs.
Exercise
Tune the ticket router with GridSearchCV (cv=5): try tfidfvectorizer__ngram_range in [(1, 1), (1, 2)], tfidfvectorizer__sublinear_tf in [False, True] and logisticregression__C in [1, 10, 100]. Use the same make_tickets() data and split, then print the best parameters, the best cross-validated accuracy and the test accuracy.
Show solution
import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
PROBLEMS = {
"delivery": ["order not delivered yet", "parcel is delayed again", "delivery boy did not come",
"tracking shows shipped but not received", "where is my package", "kab tak aayega order",
"wrong address on delivery", "courier marked delivered but i got nothing"],
"refund": ["refund not received", "return picked up but money not credited", "paisa wapas nahi aaya",
"want to return this product", "refund status still pending", "cancelled order but no refund",
"refund amount is less", "exchange or refund for damaged item"],
"payment": ["money debited but order failed", "upi payment failed", "charged twice for one order",
"payment pending for hours", "card declined at checkout", "amount cut from bank but no order",
"emi option not working", "cash on delivery not available"],
"account": ["cannot login to my account", "otp not coming", "need to change mobile number",
"password reset link not working", "account blocked suddenly", "update my email address",
"delete my account", "profile name is wrong"],
}
def make_tickets(n=1200, seed=42):
rng = np.random.default_rng(seed)
openers = ["", "hi", "hello team", "sir", "please help", "urgent"]
closers = ["", "order id 4521", "please check", "since 3 days", "very bad service", "pls resolve asap"]
labels = rng.choice(list(PROBLEMS), n)
rows = []
for label in labels:
text = f"{rng.choice(openers)} {rng.choice(PROBLEMS[label])}"
if rng.random() < 0.15:
other = rng.choice([k for k in PROBLEMS if k != label])
text += f" and also {rng.choice(PROBLEMS[other])}"
rows.append(f"{text} {rng.choice(closers)}".strip())
return pd.DataFrame({"text": rows, "team": labels})
df = make_tickets()
X_train, X_test, y_train, y_test = train_test_split(
df["text"], df["team"], test_size=0.25, random_state=42, stratify=df["team"])
pipe = make_pipeline(TfidfVectorizer(), LogisticRegression())
grid = {"tfidfvectorizer__ngram_range": [(1, 1), (1, 2)],
"tfidfvectorizer__sublinear_tf": [False, True],
"logisticregression__C": [1, 10, 100]}
search = GridSearchCV(pipe, grid, cv=5).fit(X_train, y_train)
print("Best parameters:", search.best_params_)
print(f"Best CV accuracy: {search.best_score_:.3f}")
print(f"Test accuracy: {search.score(X_test, y_test):.3f}")Best parameters: {'logisticregression__C': 100, 'tfidfvectorizer__ngram_range': (1, 2), 'tfidfvectorizer__sublinear_tf': False}
Best CV accuracy: 0.992
Test accuracy: 0.983The search picks word pairs and a large C (weak regularisation), which suits short, consistent tickets. The best cross-validated accuracy (0.992) and the test accuracy (0.983) are close, a sign that tuning did not overfit. sublinear_tf=True, which uses the logarithm of word counts, made no difference, because tickets rarely repeat a word.
Quick Quiz
1. Why does TF-IDF give the word "order" a low weight in a shop's support tickets?
Inverse document frequency lowers the weight of words that occur in most documents.
2. What does ngram_range=(1, 2) add to the features?
Bigrams keep short phrases, which helps with negations like "not received".
3. Customers often type "refnd" and "delivry". Which vectorizer setting helps most?
Misspelt words still share most of their character n-grams with the correct spelling.
4. Where should the TfidfVectorizer be fitted?
Fitting on all text leaks the test vocabulary and IDF weights into training.
Learn Machine Learning with a mentor
Free lessons get you started. Our Data Science & Machine Learning training adds real projects and code reviews from working developers.
- Project-based learning
- Mentor code reviews
- Free counselling call