How LLMs Work: Tokens & Training

You do not need to know the mathematics of neural networks to use AI well, but a simple mental picture of how a large language model works will make you a much better user. It explains why clear prompts help, why answers change from one try to the next, and why a confident answer can still be wrong.

This lesson explains, in plain words, how an LLM reads text as tokens, how it is trained, and how it writes an answer one small piece at a time.

Text becomes tokens

An LLM does not read letters or whole words. It first breaks your text into tokens: common words, parts of longer words, punctuation marks and spaces. Each token has an ID number, and the model works only with those numbers. The tool that does the splitting is called a tokenizer.

For example, a short common word like "train" is usually one token, while a rare word like "Thiruvananthapuram" may be split into several pieces. Punctuation such as a full stop or a comma is often a token of its own.

Tokens matter to you for three practical reasons:

• Limits are counted in tokens: how much text the model can read at once, and how long its reply can be.
• Cost of using AI through an API is charged per token, separately for input and output.
• Speed depends on how many tokens the model must read and write.

Every model family has its own tokenizer, so the same sentence gives different token counts on different models. For English, one token is roughly half a word to a whole word. Anthropic, for example, says 1 million tokens is roughly 555,000 English words on its current tokenizer. For exact numbers, use the provider's token-counting tool. For quick planning, a word-based estimate is enough:

import re

def pieces(text):
    """Split into words and punctuation, a rough stand-in for tokens."""
    return re.findall(r"\w+|[^\w\s]", text)

def estimate_tokens(text, words_per_token):
    words = len(text.split())
    return round(words / words_per_token)

notice = ("The Rohtak to Delhi passenger train will leave at 7:40 am from platform 2. "
          "Passengers with reserved tickets should board coach S4.")

print("Words:", len(notice.split()))
print("Words and punctuation pieces:", len(pieces(notice)))
print("Estimate at 0.75 words per token:", estimate_tokens(notice, 0.75))
print("Estimate at 0.55 words per token:", estimate_tokens(notice, 0.55))

Words: 22
Words and punctuation pieces: 26
Estimate at 0.75 words per token: 29
Estimate at 0.55 words per token: 40

The two estimates differ quite a lot, which is exactly the point: always treat a word-based count as a rough range, not an exact figure.

How an LLM is trained

Training happens in stages, long before you ever type a prompt.

1. Pretraining. The model reads an enormous collection of text: books, websites, articles and code. Its only task is to guess the next token, check the real answer, and adjust its internal numbers (called weights or parameters) a tiny bit. After billions of such guesses, the weights capture grammar, facts, writing styles and reasoning patterns.
2. Instruction tuning. The model is trained further on examples of instructions with good answers, so it learns to follow requests instead of just continuing text.
3. Feedback tuning. People (and sometimes other models) compare answers and mark which ones are better: more helpful, more honest, safer. The model is adjusted towards the preferred answers.

Two important results follow from this:

• The model does not keep a library of documents to look things up in. Its knowledge is spread across billions of numbers, a bit like how you remember the gist of many books without being able to recite a page. That is why it can mix up details.
• Training data is collected up to a knowledge cut-off date. Without a web search tool, the model knows nothing about later events.

Writing an answer, one token at a time

When you send a prompt, the model reads all of it and calculates a probability for every possible next token. It picks one, adds it to the text, and repeats, usually hundreds or thousands of times, until the answer is complete. Your phone keyboard's word suggestions work in a similar spirit, but an LLM looks at far more context and is enormously larger.

Because the model picks from probabilities, the same prompt can lead to different answers. The demo below fakes the probabilities for the word after "The train is" and draws five answers, three times over:

import random

next_word = {"late": 0.55, "on time": 0.25, "full": 0.15, "cancelled": 0.05}

for attempt in range(1, 4):
    rng = random.Random(attempt)
    picks = rng.choices(list(next_word), weights=list(next_word.values()), k=5)
    print(f"Attempt {attempt}:", ", ".join(picks))

Attempt 1: late, full, on time, late, late
Attempt 2: cancelled, full, late, late, full
Attempt 3: late, late, late, on time, on time

"late" wins most often because it is the most likely word, but less likely words still appear sometimes. A real model does this for every token of the answer, so small differences early on can lead to quite different paragraphs later.

Thinking before answering

Many newer models are reasoning models. Before writing the visible answer, they generate a hidden chain of working, called thinking or reasoning, where they plan, try approaches and check their work. This makes them much better at maths, coding and multi-step problems. The thinking still consists of tokens, so these models can take longer and cost more per request. Some tools show a short summary of the thinking; most hide the raw text.

Why LLMs make mistakes

With the training and generation steps in mind, the common failures make sense:

You will learn how to reduce these problems in Hallucinations & Verifying Answers.

Common mistakes

• Thinking the model "searches its memory" like a database. It does not look up stored pages. Give it the documents you want it to use.
• Assuming one token equals one word. Token counts vary by model and language, so leave a safety margin.
• Expecting the same answer every time. Small changes are normal. Ask for a fixed format when you need consistent results.
• Trusting arithmetic in a long answer. Recheck numbers with a calculator, a spreadsheet or code.

Key points

• LLMs read and write tokens, which are words, word pieces and punctuation.
• Limits, cost and speed are all measured in tokens, and counts differ between models.
• Training has stages: pretraining on huge text, instruction tuning and feedback tuning.
• The model generates one token at a time by picking from probabilities, so answers vary.
• Reasoning models think before answering, which helps hard problems but uses more tokens.

def token_range(text):
    words = len(text.split())
    return round(words / 0.75), round(words / 0.55)

notice = ("The annual tech fest of our Pune college starts on Friday at 10 am. "
          "Students must carry their ID cards at all times. "
          "Registration for the coding contest closes on Wednesday evening.")

low, high = token_range(notice)
print(f"about {low} to {high} tokens")

about 43 to 58 tokens