Text Classification: Naive Bayes & Logistic Regression

By the end of this lesson you will be able to build a text classifier that labels messages as spam or not spam. You will work out Naive Bayes by hand to see how it thinks, then train Naive Bayes and logistic regression in scikit-learn and compare them fairly.

Text classification means giving each piece of text a label from a fixed list: spam or not spam, positive or negative, delivery or refund. It is the most common NLP task in industry. The model learns from examples that already have labels, which is called supervised learning.

Think of the sorting clerk at a post office. After seeing thousands of letters, the clerk can glance at an envelope and drop it in the right bag. A classifier learns the same skill from labelled examples.

Naive Bayes by hand

Naive Bayes asks a simple question: "If this message were spam, how likely are its words? And if it were normal, how likely?" Then it picks the class that explains the words better.

It is called naive because it pretends each word appears independently of the others. That is clearly false ("credit" and "card" often come together), but the method still works surprisingly well for text.

Step 1. For each class, count how often each word appears.

Step 2. Turn counts into probabilities, adding 1 to every count so no word gets probability zero (the add-one smoothing from N-grams & Simple Language Models).

Step 3. For a new message, add the log probabilities of its words for each class, plus the log of how common the class is. The higher total wins.

import math
from collections import Counter

# Tiny invented training set
spam = ["win cash prize now", "free recharge offer click now", "you won cash"]
ham = ["call me after class", "send the notes now", "train is late"]

counts = {"spam": Counter(w for m in spam for w in m.split()),
          "ham": Counter(w for m in ham for w in m.split())}
vocab = set(counts["spam"]) | set(counts["ham"])
prior = {"spam": len(spam) / 6, "ham": len(ham) / 6}

def score(message, label):
    total_words = sum(counts[label].values())
    s = math.log(prior[label])
    for w in message.split():
        p = (counts[label][w] + 1) / (total_words + len(vocab))      # Step 2
        s += math.log(p)                                             # Step 3
    return s

for msg in ["win free cash now", "send notes after class"]:
    scores = {label: round(score(msg, label), 2) for label in ("spam", "ham")}
    print(f"{msg:<24} {scores} -> {max(scores, key=scores.get)}")

win free cash now        {'spam': -10.97, 'ham': -13.74} -> spam
send notes after class   {'spam': -14.56, 'ham': -11.66} -> ham

"win", "free" and "cash" were seen only in spam, so they push the first message towards spam. "send", "notes" and "class" push the second towards ham (normal messages). The word "now" appears in both classes, so it barely helps either side.

Naive Bayes with scikit-learn

scikit-learn's MultinomialNB does exactly this on word counts. Its alpha setting is the smoothing amount (1.0 by default, which is add-one). We put it in a pipeline with CountVectorizer.

To get more realistic practice data, we generate 200 invented SMS messages from templates. The generator is seeded, so you get the same messages every run.

import random
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split

def make_sms(n=200, seed=7):
    rng = random.Random(seed)
    spam_parts = [["congrats", "dear customer", "urgent", "hurry"],
                  ["you won", "claim your", "get free", "unlock"],
                  ["cash prize", "recharge", "lottery of rs 10 lakh", "loan approval", "gift voucher"],
                  ["click link now", "reply yes", "call now", "limited offer today"]]
    ham_parts = [["hi", "bro", "mummy", "sir", "ok"],
                 ["reaching home", "send the notes", "match at 5", "class cancelled", "train is late"],
                 ["by 7", "tomorrow", "after lunch", "please", "today"]]
    texts, labels = [], []
    for _ in range(n):
        is_spam = rng.random() < 0.4
        parts = spam_parts if is_spam else ham_parts
        texts.append(" ".join(rng.choice(p) for p in parts))
        labels.append("spam" if is_spam else "ham")
    return texts, labels

texts, labels = make_sms()
print(texts[:3], labels[:3])
X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.3, random_state=0, stratify=labels)

nb = make_pipeline(CountVectorizer(), MultinomialNB()).fit(X_train, y_train)
print("Test accuracy:", round(nb.score(X_test, y_test), 3))

new = ["congrats you won a free recharge", "bro match at 5 today", "please call now"]
for text, probs in zip(new, nb.predict_proba(new)):
    print(f"{text:<34} P(ham)={probs[0]:.2f} P(spam)={probs[1]:.2f}")

['dear customer unlock cash prize click link now', 'hi match at 5 today', 'dear customer you won cash prize limited offer today'] ['spam', 'ham', 'spam']
Test accuracy: 1.0
congrats you won a free recharge   P(ham)=0.00 P(spam)=1.00
bro match at 5 today               P(ham)=1.00 P(spam)=0.00
please call now                    P(ham)=0.07 P(spam)=0.93

The template data is very clean, so accuracy is perfect. Real SMS data is harder, but the code stays the same. Look at the last message: "please call now" mixes a polite ham word with a spam phrase. The model still says spam, but with 0.93 instead of 1.00. predict_proba shows that doubt, which is useful: you could send unsure messages to a human.

Logistic regression for text

Logistic regression takes a different approach. Instead of modelling how words are produced, it learns one weight per word. Positive weights push towards one class, negative weights towards the other. It adds the weights of the words in a message and turns the total into a probability. The Logistic Regression lesson in the Machine Learning course explains the maths.

The fair way to compare two models is cross-validation: split the data into 5 parts, train on 4 and test on 1, five times, and average the scores. Here we compare both models on messier messages, where some spam is written politely and some normal messages mention "free" or "call".

import random
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

rng = random.Random(3)
spam_words = "won prize cash lakh lottery free recharge offer claim click link urgent kyc loan".split()
ham_words = "class notes match home lunch train late bro mummy tomorrow meeting bus exam".split()
shared = "call now please today sir free reply ok".split()
texts, labels = [], []
for _ in range(300):
    is_spam = rng.random() < 0.4
    own = spam_words if is_spam else ham_words
    words = rng.sample(own, 2) + rng.sample(shared, 3)
    if rng.random() < 0.3:                       # some messages borrow a word from the other class
        words.append(rng.choice(ham_words if is_spam else spam_words))
    rng.shuffle(words)
    texts.append(" ".join(words))
    labels.append("spam" if is_spam else "ham")

models = {
    "Naive Bayes + counts": make_pipeline(CountVectorizer(), MultinomialNB()),
    "LogReg + TF-IDF": make_pipeline(TfidfVectorizer(), LogisticRegression(max_iter=1000)),
}
for name, model in models.items():
    scores = cross_val_score(model, texts, labels, cv=5, scoring="f1_macro")
    print(f"{name:<22} macro F1 = {scores.mean():.3f} (+/- {scores.std():.3f})")

Naive Bayes + counts   macro F1 = 0.975 (+/- 0.031)
LogReg + TF-IDF        macro F1 = 0.990 (+/- 0.014)

Both models do well, and the difference between them is small. That is typical: on many text datasets, Naive Bayes and logistic regression are close, so try both. Macro F1 is a score that treats both classes fairly; you will learn it properly in Evaluating NLP: F1, BLEU & ROUGE.

What did the model learn?

A linear model is easy to inspect. For logistic regression, coef_ holds one weight per word. The biggest positive weights point to spam (the second class alphabetically), the most negative ones to ham.

import numpy as np

model = models["LogReg + TF-IDF"].fit(texts, labels)
vec, clf = model[0], model[-1]
words = vec.get_feature_names_out()
order = np.argsort(clf.coef_[0])
print("Most ham-like: ", words[order[:5]].tolist())
print("Most spam-like:", words[order[-5:]].tolist())

Most ham-like:  ['mummy', 'notes', 'lunch', 'meeting', 'bus']
Most spam-like: ['won', 'lottery', 'link', 'loan', 'prize']

Run this after the previous example (it reuses models, texts and labels). Everyday words such as "mummy" and "notes" sit at the ham end, and "lottery" and "prize" at the spam end. Checking these words is the fastest way to spot a problem, such as a phone number that leaked into training data. For a deeper example with support tickets, typos and Hinglish, see Text Classification with TF-IDF.

Common mistakes

• Testing on the training data. Always hold out a test set or use cross-validation.
• Trusting accuracy on unbalanced data. If 95% of SMS are normal, a model that always says "ham" scores 95%. Use F1.
• Fitting the vectorizer before splitting. Keep it inside the pipeline.
• Believing Naive Bayes probabilities. They are often close to 0 or 1 even when the model is unsure. Use them for ranking, not as exact chances.

Key points

• Text classification assigns each text a label from a fixed list, learned from labelled examples.
• Naive Bayes compares how likely the words are under each class, assuming words are independent.
• MultinomialNB works on word counts; alpha controls smoothing.
• Logistic regression learns one weight per word and works very well with TF-IDF.
• Compare models with cross-validation and a fair metric such as macro F1.

import random
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline

def make_sms(n=200, seed=7):
    rng = random.Random(seed)
    spam_parts = [["congrats", "dear customer", "urgent", "hurry"],
                  ["you won", "claim your", "get free", "unlock"],
                  ["cash prize", "recharge", "lottery of rs 10 lakh", "loan approval", "gift voucher"],
                  ["click link now", "reply yes", "call now", "limited offer today"]]
    ham_parts = [["hi", "bro", "mummy", "sir", "ok"],
                 ["reaching home", "send the notes", "match at 5", "class cancelled", "train is late"],
                 ["by 7", "tomorrow", "after lunch", "please", "today"]]
    texts, labels = [], []
    for _ in range(n):
        is_spam = rng.random() < 0.4
        parts = spam_parts if is_spam else ham_parts
        texts.append(" ".join(rng.choice(p) for p in parts))
        labels.append("spam" if is_spam else "ham")
    return texts, labels

texts, labels = make_sms()
X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.3, random_state=0, stratify=labels)
lr = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LogisticRegression()).fit(X_train, y_train)
print("Test accuracy:", round(lr.score(X_test, y_test), 3))
print("P(spam) for 'please call now':", round(lr.predict_proba(["please call now"])[0][1], 2))

Test accuracy: 1.0
P(spam) for 'please call now': 0.6