Conversations, Tokens & Costs

Your first API call sent one question and got one answer. Real apps hold conversations: a student asks a follow-up, a customer adds details. Because the model has no memory between requests, your program must manage the conversation, and every extra message costs tokens.

In this lesson you will learn how multi-turn conversations work through an API, why costs grow with every turn, how to estimate and control costs, and how streaming makes apps feel faster.

Conversations are lists of messages

The API is stateless: each request is independent. To continue a conversation, you send the whole history every time, as a list of messages with alternating roles, "user" and "assistant". After each reply, your program appends it to the list:

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
history = []

def chat(user_text):
    history.append({"role": "user", "content": user_text})
    response = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=500,
        system="You are a helpful assistant for a bus booking app in Kerala.",
        messages=history,
    )
    reply = next(b.text for b in response.content if b.type == "text")
    history.append({"role": "assistant", "content": reply})
    return reply

print(chat("Is there an overnight bus from Kochi to Bengaluru?"))
print(chat("What should I carry for it?"))   # the model sees the first question too

The second question, "What should I carry for it?", only makes sense because the first question and answer are sent again. If you forget to append the assistant's reply, the model sees two user questions in a row and loses the thread.

Why each turn costs more

Since the history is resent every time, the input tokens grow with every turn. A 20-turn conversation does not cost 20 times one message; it costs much more, because turn 20 includes everything from turns 1 to 19. This simulation shows the effect with rough word-based token estimates:

history_tokens = 0
system_tokens = 60
total_input = total_output = 0

for turn in range(1, 11):
    question, answer = 40, 150                      # tokens per question and reply
    input_now = system_tokens + history_tokens + question
    total_input += input_now
    total_output += answer
    history_tokens += question + answer
    if turn in (1, 5, 10):
        print(f"Turn {turn:>2}: input {input_now:>5} tokens, running input total {total_input:>6}")

print(f"Ten turns: {total_input:,} input tokens and {total_output:,} output tokens")

Turn  1: input   100 tokens, running input total    100
Turn  5: input   860 tokens, running input total   2400
Turn 10: input  1810 tokens, running input total   9550
Ten turns: 9,550 input tokens and 1,500 output tokens

The tenth question alone sends over 1,800 tokens, although the user typed only about 40.

Estimating cost

Providers charge separately for input and output tokens, usually quoted in US dollars per million tokens, and output tokens cost more. Prices differ a lot between models and change often, so always check the provider's pricing page. The arithmetic itself is simple:

def cost(input_tokens, output_tokens, input_rate, output_rate):
    """Rates are dollars per million tokens."""
    return (input_tokens * input_rate + output_tokens * output_rate) / 1_000_000

# Example rates for illustration only: check current pricing before you plan a budget.
models = {"small fast model": (0.10, 0.50), "large model": (4.00, 20.00)}

conversations_per_day = 2_000
input_tokens, output_tokens = 9_550, 1_500        # one 10-turn chat, from the simulation

for name, (in_rate, out_rate) in models.items():
    per_chat = cost(input_tokens, output_tokens, in_rate, out_rate)
    print(f"{name:<17} ${per_chat:.4f} per chat, ${per_chat * conversations_per_day * 30:,.2f} per month")

small fast model  $0.0017 per chat, $102.30 per month
large model       $0.0682 per chat, $4,092.00 per month

The same app can cost about 40 times more on a large model than on a small one. That is why developers test whether a smaller model is good enough for each task. Reasoning models also produce thinking tokens, which are usually billed as output, so measure real usage from response.usage rather than guessing.

Keeping costs under control

• Trim or summarise history: keep the last few turns plus a short summary of older ones, as in Context Windows, Temperature & Model Settings.
• Choose the smallest model that does the job well, and test it on real examples.
• Set max_tokens to a sensible limit for each task.
• Keep the system prompt short and stable; it is sent with every request.
• Use prompt caching: when many requests start with the same long text (a policy document, a big system prompt), providers can cache it, and cached input is billed at a fraction of the normal price.
• Use batch processing for work that does not need an instant answer, such as classifying yesterday's 5,000 feedback forms. Anthropic, for example, charges half price for its Message Batches API.
• Set a spending limit in the provider console and log token usage per request.

Streaming replies

A long reply can take many seconds. Instead of waiting for the complete answer, streaming shows text as it is generated, the way chat apps type out their replies. With the Anthropic library:

import anthropic

client = anthropic.Anthropic()

with client.messages.stream(
    model="claude-opus-5-5",
    max_tokens=2000,
    messages=[{"role": "user", "content": "Write a 300-word story about a cricket match in Kanpur."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

Streaming does not make the model faster or cheaper, but users see the first words almost immediately, which feels much better. It also avoids time-outs on very long replies.

Rate limits and retries

Providers limit how many requests and tokens you can send per minute. If you exceed a limit, you get a rate limit error (HTTP status 429). The Anthropic and OpenAI Python libraries automatically retry such temporary errors a couple of times, waiting longer between attempts (exponential backoff). For large jobs, also slow your own loop down or use batches, rather than hammering the API.

Common mistakes

• Forgetting to append the assistant's reply to the history, so follow-up questions lose context.
• Letting history grow forever. Costs and delays rise every turn; trim or summarise.
• Using the largest model for everything. Test cheaper models on your real tasks.
• No usage logging. Record input_tokens and output_tokens so a sudden cost spike is easy to trace.

Key points

• APIs are stateless: send the full message history, alternating user and assistant roles, every time.
• Input tokens grow with every turn, so long conversations cost far more than the sum of their messages.
• Cost = input tokens × input rate + output tokens × output rate, with rates per million tokens.
• Control costs with history trimming, smaller models, max_tokens, prompt caching, batches and spending limits.
• Streaming shows replies as they are generated; libraries retry rate-limit errors with backoff.

history_tokens = 0
system_tokens = 60
total_input = 0

for turn in range(1, 11):
    question, answer = 40, 150
    total_input += system_tokens + history_tokens + question
    history_tokens += question + answer
    if history_tokens > 600:
        history_tokens = 120                       # replace old turns with a short summary

print(f"Capped total input: {total_input:,} tokens")
print(f"Saving compared with 9,550: {1 - total_input / 9_550:.0%}")

Capped total input: 4,000 tokens
Saving compared with 9,550: 58%