Naive Bayes

Naive Bayes classifiers are some of the fastest and simplest models in machine learning, and they have filtered email spam for decades. They work by asking: "Given what I see in this message, how likely is each class?" They answer with a centuries-old rule from probability, plus one bold simplification.

In this lesson you will apply Bayes' rule by hand, classify wines with GaussianNB, build an SMS spam filter with MultinomialNB, and learn why smoothing is essential.

Bayes' rule in plain words

Bayes' rule turns around a conditional probability. It is easy to count how often spam messages contain the word "KYC". What we actually want is the reverse: if a message contains "KYC", how likely is it to be spam?

P(spam | KYC) = P(KYC | spam) × P(spam) / P(KYC)

• P(spam) is the prior: how common spam is before we read the message.
• P(KYC | spam) is the likelihood: how often spam contains the word.
• P(spam | KYC) is the posterior: our updated belief after seeing the word.

p_spam = 0.20             # 20% of all SMS are spam
p_kyc_given_spam = 0.40   # 40% of spam SMS mention "KYC"
p_kyc_given_ham = 0.01    # 1% of normal SMS mention "KYC"

p_kyc = p_kyc_given_spam * p_spam + p_kyc_given_ham * (1 - p_spam)
p_spam_given_kyc = p_kyc_given_spam * p_spam / p_kyc

print(f"P(message mentions KYC)        = {p_kyc:.3f}")
print(f"P(spam | message mentions KYC) = {p_spam_given_kyc:.3f}")

P(message mentions KYC)        = 0.088
P(spam | message mentions KYC) = 0.909

Seeing one word raised the chance of spam from 20% to about 91%. A real message contains many words, and each one updates the belief again.

Why "naive"?

To combine many features, the classifier multiplies their likelihoods together, as if each feature were independent of the others once you know the class. That is the "naive" assumption, and it is almost never true: "lottery" and "prize" tend to appear together. Yet the model usually still ranks the classes correctly, because it only needs the right class to come out on top, not exact probabilities. In return you get a model that trains in one pass over the data, needs very little data, and copes with thousands of features.

GaussianNB for measurements

For continuous features such as lengths or chemical levels, GaussianNB assumes that each feature follows a bell-shaped (normal, or Gaussian) curve within each class. Training just computes a mean and a variance per feature per class.

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB

wine = load_wine()
X_train, X_test, y_train, y_test = train_test_split(
    wine.data, wine.target, test_size=0.25, random_state=42, stratify=wine.target)

nb = GaussianNB().fit(X_train, y_train)
print(f"Test accuracy: {nb.score(X_test, y_test):.3f}")
print("Class priors:", nb.class_prior_.round(3))
col = wine.feature_names.index("alcohol")
print("Mean alcohol per class:", nb.theta_[:, col].round(2))
print("Probabilities for the first test wine:", nb.predict_proba(X_test[:1]).round(3))

Test accuracy: 0.978
Class priors: [0.331 0.398 0.271]
Mean alcohol per class: [13.71 12.27 13.08]
Probabilities for the first test wine: [[1. 0. 0.]]

The priors are simply the class shares in the training data, and theta_ holds the learned means. Notice the probability of exactly 1.0: naive Bayes is often overconfident, because multiplying 13 "independent" pieces of evidence exaggerates certainty. Trust its ranking more than its exact probabilities. Naive Bayes does not need feature scaling, since each feature gets its own mean and variance.

MultinomialNB for text: an SMS spam filter

For text, the features are word counts. CountVectorizer turns each message into a row of counts, one column per word in the vocabulary. MultinomialNB then learns how often each word appears in each class. Here are 16 short SMS messages, 7 of them spam, in English and Hinglish.

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

messages = [
    "Your KYC is pending, update now or your account will be blocked",
    "Congratulations! You won a lottery of Rs 25 lakh, click the link to claim",
    "Dear customer your KYC update is required, click link immediately",
    "You have won free recharge, claim your prize now",
    "Urgent: account blocked, share OTP to unblock",
    "Lottery winner! Send your bank details to claim prize",
    "Free gift voucher waiting, click now",
    "Bhai kal match dekhne chalein?",
    "Mummy, I will reach Jaipur by 7 pm",
    "Your OTP for login is 482913. Do not share it with anyone",
    "Meeting moved to 3 pm tomorrow",
    "Kya tum notes bhej sakte ho for the exam?",
    "Your order has been delivered. Thank you for shopping",
    "Happy birthday Priya! Party at Kabir's place tonight",
    "Can you pick up milk on the way home",
    "Train is running 30 minutes late, see you at the station",
]
labels = [1] * 7 + [0] * 9   # 1 = spam

vec = CountVectorizer()
X = vec.fit_transform(messages)
model = MultinomialNB(alpha=1.0).fit(X, labels)
print("Vocabulary size:", len(vec.vocabulary_))

new = ["Click the link to claim your free prize",
       "Are we meeting for the match tomorrow?",
       "Update KYC now or account blocked",
       "Your OTP is 551203, do not share it"]
for text, p in zip(new, model.predict_proba(vec.transform(new))[:, 1]):
    print(f"P(spam) = {p:.2f}  {text}")

words = vec.get_feature_names_out()
spamminess = model.feature_log_prob_[1] - model.feature_log_prob_[0]
print("Most spam-like words:", list(words[np.argsort(spamminess)[::-1][:6]]))

Vocabulary size: 99
P(spam) = 1.00  Click the link to claim your free prize
P(spam) = 0.02  Are we meeting for the match tomorrow?
P(spam) = 1.00  Update KYC now or account blocked
P(spam) = 0.21  Your OTP is 551203, do not share it
Most spam-like words: ['click', 'claim', 'now', 'update', 'link', 'kyc']

The filter flags the prize and KYC messages and lets the friendly ones through. The genuine OTP message scores much higher than the other genuine message because "OTP" and "share" also appear in spam, a realistic weakness with so little data. feature_log_prob_ holds the learned (log) word probabilities per class; comparing the two rows shows which words push a message towards spam. CountVectorizer lowercases text and splits it into words for you, and its vocabulary comes only from the training messages, so new words are simply ignored.

Smoothing: why alpha matters

Suppose the word "tonight" never appeared in any spam message in training. Without correction, P("tonight" | spam) would be exactly 0, and because the model multiplies likelihoods, one such word would force the spam probability to 0 no matter how many other spam words the message had. Laplace smoothing fixes this by adding a small count, alpha, to every word in every class. The default alpha=1.0 works well; smaller values trust the data more, larger values smooth more. Treat alpha as a hyperparameter to tune.

The naive Bayes family

Common mistakes

• Using GaussianNB on word counts, or MultinomialNB on negative numbers. Match the variant to the kind of feature; MultinomialNB needs non-negative counts.
• Fitting the vectorizer on all messages, including test ones. Fit CountVectorizer on the training text only, or put it in a pipeline.
• Taking the probabilities literally. Naive Bayes is usually overconfident; use it for ranking, or calibrate it.
• Setting alpha=0. Unseen words then create zero probabilities; keep some smoothing.

Key points

• Bayes' rule updates a prior belief with the evidence: posterior is proportional to likelihood times prior.
• Naive Bayes assumes features are independent within each class; the assumption is wrong but the model often works well.
• GaussianNB suits continuous features; MultinomialNB and ComplementNB suit word counts.
• CountVectorizer turns text into word counts; alpha smoothing prevents zero probabilities.
• Naive Bayes is very fast, needs little data and makes a strong baseline for text, but its probabilities are overconfident.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import ComplementNB
from sklearn.pipeline import make_pipeline

messages = [
    "Your KYC is pending, update now or your account will be blocked",
    "Congratulations! You won a lottery of Rs 25 lakh, click the link to claim",
    "Dear customer your KYC update is required, click link immediately",
    "You have won free recharge, claim your prize now",
    "Urgent: account blocked, share OTP to unblock",
    "Lottery winner! Send your bank details to claim prize",
    "Free gift voucher waiting, click now",
    "Bhai kal match dekhne chalein?",
    "Mummy, I will reach Jaipur by 7 pm",
    "Your OTP for login is 482913. Do not share it with anyone",
    "Meeting moved to 3 pm tomorrow",
    "Kya tum notes bhej sakte ho for the exam?",
    "Your order has been delivered. Thank you for shopping",
    "Happy birthday Priya! Party at Kabir's place tonight",
    "Can you pick up milk on the way home",
    "Train is running 30 minutes late, see you at the station",
]
labels = [1] * 7 + [0] * 9

spam_filter = make_pipeline(CountVectorizer(), ComplementNB()).fit(messages, labels)
new = ["Claim your lottery prize now", "Kabir, the train is late",
       "Share OTP to unblock your account", "Party tonight at my place"]
for text, label in zip(new, spam_filter.predict(new)):
    print(f"{'spam' if label else 'ok':<5} {text}")

spam  Claim your lottery prize now
ok    Kabir, the train is late
spam  Share OTP to unblock your account
ok    Party tonight at my place