Tokenisation: Words & Sentences

By the end of this lesson you will be able to split text into words and sentences correctly, including tricky cases such as prices, links, short forms and abbreviations. You will also see how scikit-learn and spaCy do it for you.

Tokenisation means cutting text into small units called tokens. Usually a token is a word or a punctuation mark. Every NLP method you will learn works on tokens, so getting this step right matters.

Think of a long train arriving at a station. Before anyone can count passengers, the train has to be split into coaches. Tokenisation splits text into coaches that later steps can count and study.

Why split() is not enough

Python's str.split() cuts text at spaces. It is a good start, but punctuation stays stuck to the words.

Step 1. Split a message with split().

Step 2. Look at the tokens. Do "late," and "late" count as the same word?

msg = "Delivery is late, again! Order ID 4521 costs ₹1,299. Don't cancel it."
print(msg.split())

['Delivery', 'is', 'late,', 'again!', 'Order', 'ID', '4521', 'costs', '₹1,299.', "Don't", 'cancel', 'it.']

We get "late," with a comma, "again!" with an exclamation mark and "it." with a full stop. To a computer, "late," and "late" are different strings, so counts get split.

A regex tokeniser

A better approach is to describe what a token looks like with a regular expression and use re.findall(). The pattern below is a list of alternatives separated by |. The engine tries them in order, so put the most specific shapes first.

import re

TOKEN = re.compile(r"""
    https?://\S+              # links
  | ₹\s?\d[\d,]*(?:\.\d+)?    # rupee amounts
  | \d+(?:[.,]\d+)*           # numbers
  | \w+(?:'\w+)?              # words, with an optional apostrophe part
  | [^\w\s]                   # any single punctuation mark
""", re.VERBOSE)

msg = "Delivery is late, again! Order ID 4521 costs ₹1,299. Don't cancel it. See https://shop.example/t/4521"
print(TOKEN.findall(msg))

['Delivery', 'is', 'late', ',', 'again', '!', 'Order', 'ID', '4521', 'costs', '₹1,299', '.', "Don't", 'cancel', 'it', '.', 'See', 'https://shop.example/t/4521']

Now punctuation marks are separate tokens, "₹1,299" stays together as one price, "Don't" stays as one word and the link is kept whole. You decide the rules for your data. A tokeniser for tweets would also keep hashtags such as "#IndVsAus" together.

Tokens and vocabulary

Two words you will see often:

• Token: one occurrence of a word in the text.
• Type: one distinct word. The set of all types is the vocabulary.

The sentence "the train left and the bus left" has 7 tokens but only 5 types. Vocabulary size matters, because many models create one column per type.

from collections import Counter

text = "the train left and the bus left and the auto stayed"
tokens = text.split()
vocab = Counter(tokens)
print("Tokens:", len(tokens))
print("Types :", len(vocab))
print(vocab.most_common(3))

Tokens: 11
Types : 7
[('the', 3), ('left', 2), ('and', 2)]

Splitting sentences

Many tasks need sentences first: summarisation picks whole sentences, and translation works sentence by sentence. Splitting at every full stop fails on abbreviations such as "Dr.", "Rs." and "No.".

Step 1. Split after ., ! or ? when a space and a capital letter follow.

Step 2. Glue back pieces that end with a known abbreviation.

import re

ABBREVIATIONS = {"dr.", "mr.", "mrs.", "rs.", "no.", "st."}

def split_sentences(text):
    parts = re.split(r"(?<=[.!?])\s+(?=[A-Z])", text)
    sentences = []
    for part in parts:
        if sentences and sentences[-1].split()[-1].lower() in ABBREVIATIONS:
            sentences[-1] += " " + part         # previous piece ended with an abbreviation
        else:
            sentences.append(part)
    return sentences

text = "Dr. Meera Iyer paid Rs. 500 at the clinic. The receipt was not sent! Can you check?"
for s in split_sentences(text):
    print("-", s)

- Dr. Meera Iyer paid Rs. 500 at the clinic.
- The receipt was not sent!
- Can you check?

The (?<=[.!?]) part is a lookbehind: it checks the previous character without using it up, so the punctuation stays with its sentence. Without the abbreviation list, "Dr." and "Rs." would wrongly end sentences.

How libraries tokenise

You rarely write a tokeniser from scratch in real projects. scikit-learn's vectorizers use the default pattern (?u)\b\w\w+\b, which keeps runs of two or more word characters. That has a surprising side effect: one-letter words like "I" and "a" disappear.

from sklearn.feature_extraction.text import CountVectorizer

analyse = CountVectorizer().build_analyzer()
print(analyse("I got a refund of ₹500, Don't worry!"))

['got', 'refund', 'of', '500', 'don', 'worry']

Also notice "Don't" became "don" (the "t" is too short to keep). For most topic tasks this is fine. If you need single letters or special tokens, pass your own token_pattern or tokenizer.

spaCy has a careful rule-based tokeniser. It splits "Don't" into "Do" and "n't", keeps "₹" as its own token and knows many abbreviations.

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Dr. Meera paid ₹500. Don't worry!")
print([t.text for t in doc])
print([s.text for s in doc.sents])

['Dr.', 'Meera', 'paid', '₹', '500', '.', 'Do', "n't", 'worry', '!']
['Dr. Meera paid ₹500.', "Don't worry!"]

Common mistakes

• Using split() and forgetting punctuation. "late," and "late" become different words.
• Splitting sentences at every full stop. Abbreviations, decimals and initials break the result.
• Not knowing your library's default pattern. scikit-learn drops one-letter tokens; check with build_analyzer().
• Tokenising training and test text differently. Use one function for both.

Key points

• Tokenisation splits text into tokens, usually words and punctuation marks.
• str.split() leaves punctuation attached; a regex tokeniser gives you control.
• Tokens are occurrences, types are distinct words, and the vocabulary is the set of types.
• Sentence splitting must handle abbreviations such as "Dr." and "Rs.".
• scikit-learn's default token_pattern drops one-letter tokens; spaCy has a richer rule-based tokeniser.

import re

TOKEN = re.compile(r"""
    [#@]\w+                   # hashtags and mentions first
  | https?://\S+
  | \d+(?:[.,]\d+)*
  | \w+(?:'\w+)?
  | [^\w\s]
""", re.VERBOSE)

print(TOKEN.findall("Great win! #IndVsAus thanks @ananya_k for the tickets"))

['Great', 'win', '!', '#IndVsAus', 'thanks', '@ananya_k', 'for', 'the', 'tickets']