Topic Modelling

By the end of this lesson you will be able to discover the main themes in a collection of documents without any labels. You will run two topic models in scikit-learn, LDA and NMF, read their top words, and see what mix of topics each document contains.

So far, every classifier needed labelled examples. But often you have a big pile of text and no labels at all: 5,000 feedback forms from a college, or a year of customer emails. You want to know what people talk about. Topic modelling finds groups of words that tend to appear together and calls each group a topic. This is unsupervised learning: the model gets no labels.

Think of sorting a heap of newspaper cuttings on the floor. Without reading every one carefully, you notice some are full of "wicket", "runs" and "over", others of "rain", "monsoon" and "flood". You make piles. A topic model makes those piles for you.

The idea behind LDA

Latent Dirichlet Allocation (LDA) is the classic topic model. It imagines that every document is written like this:

1. Pick a mix of topics for the document, for example 70% cricket and 30% weather.
2. For each word, pick a topic from that mix, then pick a word that is common in that topic.

LDA works backwards from the real documents to find the topics (lists of word probabilities) and each document's mix. "Latent" means hidden: nobody told the model what the topics are.

LDA works on word counts, so we use CountVectorizer, remove stop words (this is a topic task) and drop very rare words.

Running LDA

Step 1. Prepare a small invented set of short news snippets about three themes.

Step 2. Turn them into counts.

Step 3. Fit LDA with 3 topics and print the top words of each.

import numpy as np
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer

docs = [  # invented news snippets
    "india won the cricket match by five wickets", "the batter scored a century in the test match",
    "the bowler took three wickets in the final over", "fans cheered as the captain hit a six",
    "the cricket team will tour australia next month", "rain stopped the cricket match in kolkata",
    "heavy monsoon rain flooded streets in mumbai", "the weather office predicts more rain this week",
    "the monsoon reached kerala early this year", "farmers wait for monsoon rain to sow crops",
    "flood warning issued as river level rises", "hot weather and heatwave hit north india",
    "the sensex rose as bank shares gained", "investors bought shares after strong company results",
    "the rupee fell against the dollar today", "the bank raised interest rates on loans",
    "markets fell as oil prices rose", "the company reported higher profit this quarter",
]

vec = CountVectorizer(stop_words="english", min_df=2)
counts = vec.fit_transform(docs)
words = vec.get_feature_names_out()

lda = LatentDirichletAllocation(n_components=3, max_iter=50, random_state=2)
doc_topics = lda.fit_transform(counts)

for i, weights in enumerate(lda.components_):
    top = np.argsort(weights)[::-1][:5]
    print(f"Topic {i}:", ", ".join(words[top]))

Topic 0: shares, rose, bank, company, fell
Topic 1: rain, monsoon, weather, hit, india
Topic 2: match, cricket, wickets, india, rain

Look at the topic word lists and give each a name yourself: the model only gives numbers. This naming step is always human work. max_iter=50 gives the model more passes over the data than the default 10. With only 18 short documents, the result also depends on random_state: other seeds mix cricket and business words in one topic. With thousands of documents, topics become much more stable.

Each document's topic mix

fit_transform also returned a row per document: its mix of topics, which adds up to 1. You can use it to find the main topic of each document, or to tag new documents.

import numpy as np
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer

docs = [
    "india won the cricket match by five wickets", "the batter scored a century in the test match",
    "the bowler took three wickets in the final over", "fans cheered as the captain hit a six",
    "the cricket team will tour australia next month", "rain stopped the cricket match in kolkata",
    "heavy monsoon rain flooded streets in mumbai", "the weather office predicts more rain this week",
    "the monsoon reached kerala early this year", "farmers wait for monsoon rain to sow crops",
    "flood warning issued as river level rises", "hot weather and heatwave hit north india",
    "the sensex rose as bank shares gained", "investors bought shares after strong company results",
    "the rupee fell against the dollar today", "the bank raised interest rates on loans",
    "markets fell as oil prices rose", "the company reported higher profit this quarter",
]
vec = CountVectorizer(stop_words="english", min_df=2)
counts = vec.fit_transform(docs)
lda = LatentDirichletAllocation(n_components=3, max_iter=50, random_state=2).fit(counts)

new = ["rain may stop the cricket match", "bank shares rose after the monsoon"]
for text, mix in zip(new, lda.transform(vec.transform(new))):
    print(f"{text:<36}", np.round(mix, 2))

rain may stop the cricket match      [0.08 0.12 0.8 ]
bank shares rose after the monsoon   [0.67 0.26 0.07]

Each new snippet gets a probability for every topic, in the order Topic 0, 1, 2. The first is mostly cricket (0.80) with a little weather. The second is mostly business, but "monsoon" gives it a real share of the weather topic (0.26). A classifier must pick one label; a topic model reports the mix.

NMF: another way to find topics

Non-negative matrix factorisation (NMF) splits the document-word table into two smaller tables: documents × topics and topics × words, with no negative numbers. It usually works on TF-IDF and often gives cleaner topics on short texts such as tweets and tickets.

import numpy as np
from sklearn.decomposition import NMF
from sklearn.feature_extraction.text import TfidfVectorizer

docs = [
    "india won the cricket match by five wickets", "the batter scored a century in the test match",
    "the bowler took three wickets in the final over", "fans cheered as the captain hit a six",
    "the cricket team will tour australia next month", "rain stopped the cricket match in kolkata",
    "heavy monsoon rain flooded streets in mumbai", "the weather office predicts more rain this week",
    "the monsoon reached kerala early this year", "farmers wait for monsoon rain to sow crops",
    "flood warning issued as river level rises", "hot weather and heatwave hit north india",
    "the sensex rose as bank shares gained", "investors bought shares after strong company results",
    "the rupee fell against the dollar today", "the bank raised interest rates on loans",
    "markets fell as oil prices rose", "the company reported higher profit this quarter",
]
vec = TfidfVectorizer(stop_words="english", min_df=2)
tfidf = vec.fit_transform(docs)
words = vec.get_feature_names_out()

nmf = NMF(n_components=3, random_state=0)
doc_topics = nmf.fit_transform(tfidf)
for i, weights in enumerate(nmf.components_):
    top = np.argsort(weights)[::-1][:5]
    print(f"Topic {i}:", ", ".join(words[top]))
print("Main topic per document:", doc_topics.argmax(axis=1).tolist())

Topic 0: monsoon, rain, weather, hit, wickets
Topic 1: match, cricket, wickets, india, rain
Topic 2: fell, company, shares, rose, bank
Main topic per document: [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 2, 2, 2, 2, 2, 2]

The argmax line shows the strongest topic for each of the 18 documents, in order. Compare it with the three groups of six in the list: you can see how well the unsupervised model recovered the themes without ever seeing a label.

Choosing the number of topics

Topic models need you to choose n_components. There is no single right answer. Practical tips:

• Try several values (for example 5, 10, 15, 20) and read the top words of each. Pick the one where topics are clear and not repeated.
• LDA in scikit-learn has a perplexity() method: lower is better on held-out documents, but it does not always match what humans find meaningful.
• Topic coherence scores measure whether a topic's top words really appear together; libraries such as gensim compute them.

Common mistakes

• Keeping stop words. Topics fill up with "the", "is" and "was". Remove them for topic modelling.
• Feeding TF-IDF to LDA. LDA expects counts. Use counts for LDA and TF-IDF for NMF.
• Treating topic numbers as names. "Topic 2" means nothing until a human reads its words.
• Trusting topics from very few documents. Small data gives unstable topics; change random_state and see if they hold.

Key points

• Topic modelling finds themes in unlabelled documents; it is unsupervised.
• LDA sees each document as a mix of topics, and each topic as a mix of words.
• Use CountVectorizer with LDA and TfidfVectorizer with NMF, both with stop words removed.
• components_ holds topic-word weights; transform gives each document's topic mix.
• Choose the number of topics by reading the topics, helped by perplexity or coherence.

import numpy as np
from sklearn.decomposition import NMF
from sklearn.feature_extraction.text import TfidfVectorizer

docs = [
    "india won the cricket match by five wickets", "the batter scored a century in the test match",
    "the bowler took three wickets in the final over", "fans cheered as the captain hit a six",
    "the cricket team will tour australia next month", "rain stopped the cricket match in kolkata",
    "heavy monsoon rain flooded streets in mumbai", "the weather office predicts more rain this week",
    "the monsoon reached kerala early this year", "farmers wait for monsoon rain to sow crops",
    "flood warning issued as river level rises", "hot weather and heatwave hit north india",
    "the sensex rose as bank shares gained", "investors bought shares after strong company results",
    "the rupee fell against the dollar today", "the bank raised interest rates on loans",
    "markets fell as oil prices rose", "the company reported higher profit this quarter",
]
vec = TfidfVectorizer(stop_words="english", min_df=2)
tfidf = vec.fit_transform(docs)
words = vec.get_feature_names_out()
for k in (2, 4):
    nmf = NMF(n_components=k, random_state=0).fit(tfidf)
    print(f"k={k}")
    for i, w in enumerate(nmf.components_):
        print(f"  Topic {i}:", ", ".join(words[np.argsort(w)[::-1][:4]]))

k=2
  Topic 0: monsoon, rain, weather, hit
  Topic 1: match, cricket, wickets, india
k=4
  Topic 0: monsoon, rain, weather, hit
  Topic 1: match, cricket, wickets, india
  Topic 2: company, shares, bank, rose
  Topic 3: fell, rose, bank, shares