Pretrained Models & Hugging Face

By the end of this lesson you will know what a pretrained model is, how to find a suitable one on the Hugging Face Hub, and how to use it with a few lines of Python. You will also understand what happens inside: token IDs, attention masks, logits and labels.

Training a transformer from scratch needs huge amounts of text and expensive computers. Almost nobody does it for a single project. Instead, you start from a model that someone has already trained on a large amount of text and reuse it. This is called transfer learning.

Think of hiring a fresh graduate in Pune. You do not teach them English or arithmetic; they already know that. You only teach them your company's products and rules. A pretrained model is that graduate: it already knows a lot about language, and you only teach it your task.

How models are pretrained

Pretraining is self-supervised: the labels come from the text itself, so no human labelling is needed.

• Masked language modelling (BERT-style): hide some words and train the model to guess them. "My [MASK] has not arrived" should give "order" or "parcel".
• Next-word prediction (GPT-style): train the model to predict each next word from the words before it.

After reading billions of sentences this way, the model has learned grammar, word meanings and many facts. It has also learned the biases and errors in that text, which you will study in Bias & Responsible NLP.

The Hugging Face Hub

The Hugging Face Hub (huggingface.co) is a large public collection of models and datasets. The transformers library downloads models from it by name, such as google-bert/bert-base-uncased.

Before you use a model, read its model card, the page that describes it. Check:

For Indian languages, look at models trained on Indian text, such as Google's MuRIL and AI4Bharat's IndicBERT, and test them on your own data.

The fastest way: pipelines

A pipeline wraps a tokenizer, a model and the post-processing for one task. This is plain code: it needs transformers and torch on your computer and downloads the model the first time.

from transformers import pipeline

sentiment = pipeline("sentiment-analysis", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
print(sentiment("The delivery was quick and the packing was neat"))

ner = pipeline("ner", model="dslim/bert-base-NER", aggregation_strategy="simple")
print(ner("Aarav works at Infosys in Pune"))

zero_shot = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
print(zero_shot("My money was debited but the order failed",
                candidate_labels=["delivery", "refund", "payment", "account"]))

Each call returns Python lists and dictionaries. The sentiment pipeline returns a label and a score for each text. The NER pipeline returns one dictionary per entity with its entity_group, word, score, start and end. The zero-shot pipeline returns the labels sorted from most to least likely with matching scores. Exact scores depend on the model version, so we do not print them here; run the code to see them.

Zero-shot classification is worth noticing: it sorts text into labels you choose at run time, with no training examples at all. It is a quick way to build a first ticket router, which you can then compare with a trained one.

What a tokenizer returns

Every pipeline starts with the model's own tokenizer (see Subword Tokenisation & BPE). Calling it on a batch of texts returns input IDs (token numbers) and an attention mask. This part runs without PyTorch, so we can show real output.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
batch = tokenizer(["Refund not received", "My UPI payment failed twice"], padding=True)
print(batch["input_ids"])
print(batch["attention_mask"])
print(tokenizer.convert_ids_to_tokens(batch["input_ids"][0]))

[[101, 25416, 8630, 2025, 2363, 102, 0, 0], [101, 2026, 2039, 2072, 7909, 3478, 3807, 102]]
[[1, 1, 1, 1, 1, 1, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1]]
['[CLS]', 'ref', '##und', 'not', 'received', '[SEP]', '[PAD]', '[PAD]']

Three things to notice:

• BERT adds special tokens: [CLS] (ID 101) at the start and [SEP] (ID 102) at the end.
• Texts in one batch must have the same length, so the shorter one is filled with [PAD] (ID 0). This is padding.
• The attention mask is 1 for real tokens and 0 for padding, so the model ignores the padding.

The same steps in plain Python

Here is a toy version of what a tokenizer does, so you can run and change it in the browser.

Step 1. Map words to IDs, with special IDs for padding, unknown words, start and end.

Step 2. Add [CLS] and [SEP], then pad every sequence to the longest one.

Step 3. Build the attention mask.

vocab = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3,
         "refund": 4, "not": 5, "received": 6, "upi": 7, "payment": 8, "failed": 9}

def encode_batch(texts):
    ids = [[2] + [vocab.get(w, 1) for w in t.lower().split()] + [3] for t in texts]   # Steps 1-2
    longest = max(len(seq) for seq in ids)
    padded = [seq + [0] * (longest - len(seq)) for seq in ids]
    mask = [[1 if i != 0 else 0 for i in seq] for seq in padded]                       # Step 3
    return padded, mask

ids, mask = encode_batch(["Refund not received", "My UPI payment failed twice"])
print(ids)
print(mask)

[[2, 4, 5, 6, 3, 0, 0], [2, 1, 7, 8, 9, 1, 3]]
[[1, 1, 1, 1, 1, 0, 0], [1, 1, 1, 1, 1, 1, 1]]

"My" and "twice" are not in our tiny vocabulary, so they become [UNK] (ID 1). Real subword tokenizers avoid most unknowns, as you saw above.

From logits to labels

The model's last layer outputs logits: one raw score per class. The pipeline turns them into probabilities with softmax and looks up the label names in the model's id2label setting. You can reproduce that last step yourself.

import math

id2label = {0: "NEGATIVE", 1: "POSITIVE"}
logits = [-1.8, 2.6]                     # example raw scores from a model's last layer

exps = [math.exp(x) for x in logits]
probs = [e / sum(exps) for e in exps]
best = max(range(len(probs)), key=probs.__getitem__)
print([round(p, 3) for p in probs])
print({"label": id2label[best], "score": round(probs[best], 3)})

[0.012, 0.988]
{'label': 'POSITIVE', 'score': 0.988}

That dictionary has the same shape as a sentiment pipeline's output. Nothing magical happens at the end of a pipeline: softmax and a lookup.

Common mistakes

• Not pinning the model name. Always pass model=...; the default can change between library versions.
• Mixing a tokenizer and model from different checkpoints. Load both from the same name.
• Skipping the model card. Language, licence and training data decide whether a model fits.
• Feeding very long texts. Many encoder models accept at most 512 tokens; longer input is cut off. Split long documents.
• Trusting zero-shot labels blindly. Measure on a labelled sample before using it for real routing.

Key points

• Pretrained models learn language from huge text with self-supervised tasks, then get reused.
• The Hugging Face Hub hosts models; read the model card for task, languages, data, licence and limits.
• pipeline(task, model=...) gives working sentiment, NER, zero-shot and other tools in a few lines.
• Tokenizers return input IDs and attention masks, with special tokens and padding.
• Models output logits; softmax and id2label turn them into labels and scores.

vocab = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3,
         "refund": 4, "not": 5, "received": 6, "upi": 7, "payment": 8, "failed": 9}

def encode_batch(texts, max_length):
    out = []
    for t in texts:
        words = [vocab.get(w, 1) for w in t.lower().split()][: max_length - 2]   # room for CLS and SEP
        seq = [2] + words + [3]
        out.append(seq + [0] * (max_length - len(seq)))
    mask = [[1 if i != 0 else 0 for i in seq] for seq in out]
    return out, mask

ids, mask = encode_batch(["Refund not received", "UPI payment failed refund not received"], max_length=5)
print(ids)
print(mask)

[[2, 4, 5, 6, 3], [2, 7, 8, 9, 3]]
[[1, 1, 1, 1, 1], [1, 1, 1, 1, 1]]