NLP Introduction

By the end of this lesson you will know what natural language processing (NLP) is, where you already use it every day, and why language is hard for a computer. You will also write a tiny program that reads customer messages and sorts them.

You only need basic Python: strings, lists, dictionaries and loops. Everything else is explained as we go.

What is NLP?

Natural language means the languages people speak and write: English, Hindi, Tamil, Bengali, Hinglish chat messages and so on. Natural language processing is the part of computer science that makes computers read, understand and produce this kind of text.

A computer is very good with numbers and fixed formats. A railway ticket has a train number, a date and a seat. Those fields are easy to store and compare. A complaint email is different. It has no fixed fields, it may have spelling mistakes, and the same problem can be written in a hundred ways.

NLP gives us tools to turn messy text into something a program can work with. Think of it as a very patient clerk who reads every message and fills in a form: what is this about, who is mentioned, is the person happy or angry.

NLP in daily life in India

You use NLP many times a day, often without noticing:

India adds special challenges. The Constitution's Eighth Schedule lists 22 languages, many with their own scripts. People also mix languages in one sentence and type Hindi in Roman letters ("kal milte hain"). A good Indian NLP system must handle all of this. You will learn how in Working with Indian Languages.

Why language is hard for computers

Here are four problems that make NLP interesting:

1. Ambiguity. "I saw Priya with the telescope" has two meanings. Who had the telescope?
2. Many ways to say one thing. "Order not delivered", "parcel never came" and "kab aayega mera saman" all mean the same.
3. Context. "This phone is sick!" can be praise from a teenager or a complaint from a buyer.
4. Messy input. Typos, emojis, short forms ("pls", "asap") and mixed scripts.

Your first text program

Let us start the way many real systems started: with simple rules. A small online shop in Pune gets messages from customers. We want to sort each message into a team.

Step 1. Write a few keywords for each team.

Step 2. Lowercase each message and check which keywords appear.

Step 3. Pick the team with the most keyword matches.

# Invented messages for practice
messages = [
    "My order is not delivered yet",
    "Refund not received for the cancelled order",
    "Payment failed but money debited from bank",
    "Parcel never came, please help",
]

keywords = {
    "delivery": ["delivered", "delivery", "courier", "shipped"],
    "refund": ["refund", "return", "cancelled"],
    "payment": ["payment", "debited", "upi", "card"],
}

for msg in messages:
    text = msg.lower()
    scores = {team: sum(word in text for word in words) for team, words in keywords.items()}
    best = max(scores, key=scores.get)
    if scores[best] == 0:
        best = "unknown"
    print(f"{best:<9} <- {msg}")

delivery  <- My order is not delivered yet
refund    <- Refund not received for the cancelled order
payment   <- Payment failed but money debited from bank
unknown   <- Parcel never came, please help

The first three messages go to the right team. The fourth fails: "Parcel never came" is clearly a delivery problem, but none of our keywords appear in it.

Why rules are not enough

You could add "parcel" and "came" to the list. Then a customer writes "saman nahi mila" and the rule fails again. Every new way of writing needs a new rule, and soon rules start to clash with each other.

This is why modern NLP learns from examples. You give the computer many labelled messages, and it finds the patterns itself. Rules still have a place: they are fast, cheap and easy to explain. Good systems often use both.

Here is a second tiny program. It counts which words appear most often in a set of food reviews. Counting words is the first step of almost every NLP method you will learn.

from collections import Counter

# Invented reviews from a food app in Jaipur
reviews = [
    "Tasty paneer, fast delivery",
    "Paneer was cold and delivery was late",
    "Very tasty dal, will order again",
    "Late delivery, food was cold",
]

words = []
for review in reviews:
    for word in review.lower().replace(",", " ").split():
        words.append(word)

print(Counter(words).most_common(5))

[('delivery', 3), ('was', 3), ('tasty', 2), ('paneer', 2), ('cold', 2)]

Even this simple count tells a story: delivery and cold food come up a lot. Notice that "was" is also near the top. Such common, low-meaning words are called stop words, and you will meet them in Stop Words. In the next lessons you will clean text properly, split it into tokens, and turn it into numbers that a machine learning model can use.

What you will learn in this course

The course moves from basics to advanced topics:

• Preparing text: cleaning, regular expressions, tokenisation, stop words, stemming and lemmatisation.
• Text as numbers: bag of words, TF-IDF, n-grams and word embeddings.
• Classic tasks: classification, sentiment, part-of-speech tags, named entities and topics.
• Transformers: attention, pretrained models, fine-tuning and semantic search.
• Practice: Indian languages, evaluation, responsible NLP, large language models and a final project.

Runnable examples use Python's standard library and scikit-learn. Code for spaCy and Hugging Face Transformers is shown as plain code with its output, because those libraries do not run inside the browser editor.

Common mistakes

• Thinking NLP means only chatbots. Search, spam filters, routing and translation are all NLP.
• Starting with the biggest model. A simple word-count model is often good enough and is always a useful baseline to compare against.
• Ignoring how people really write. Test your ideas on real, messy messages, not only on neat sentences.

Key points

• NLP makes computers read, understand and produce human language.
• You meet NLP in keyboards, search, chatbots, spam filters, support desks and translation.
• Language is hard because of ambiguity, many ways of saying one thing, context and messy input.
• Rule-based programs are simple but break when people write in new ways.
• Modern NLP learns patterns from labelled examples, often alongside a few rules.

messages = ["OTP not coming for login", "parcel never came"]
keywords = {
    "delivery": ["delivered", "delivery", "courier", "shipped", "parcel", "never came"],
    "refund": ["refund", "return", "cancelled"],
    "payment": ["payment", "debited", "upi", "card"],
    "account": ["login", "otp", "password"],
}
for msg in messages:
    text = msg.lower()
    scores = {team: sum(word in text for word in words) for team, words in keywords.items()}
    print(max(scores, key=scores.get), "<-", msg)

account <- OTP not coming for login
delivery <- parcel never came