Open-weight Models & Running AI Locally

ChatGPT, Claude and Gemini run on their companies' servers: you send text over the internet and pay per token. Open-weight models are different: the company publishes the model's weights (the billions of numbers learnt during training), so anyone can download the model and run it on their own laptop, office server or cloud machine.

In this lesson you will learn what "open weights" does and does not mean, why teams run models themselves, how to estimate the memory a model needs, and how to run a model locally behind the same kind of API you already know.

What "open weights" means

With an open-weight model you get the trained weights and code to run them. You usually do not get the training data, and the licence decides what you may do. Licences vary a lot: some are standard open-source licences such as Apache 2.0, while others add conditions, such as limits for very large companies or an acceptable-use policy. Read the licence before using a model in a product; "free to download" is not the same as "free for any commercial use".

Well-known families of open-weight models include Meta's Llama, Google's Gemma, Mistral's models, Alibaba's Qwen, DeepSeek's models and OpenAI's gpt-oss. New versions appear every few months, so check each project's official page for current sizes and licences. Models are usually downloaded from the Hugging Face Hub or through a tool such as Ollama.

Why run a model yourself, and why not

Reasons teams choose open-weight models:

• Data stays inside: prompts never leave your machine or your company's servers, which helps with confidential and personal data. Your duties under the DPDP Act still apply; you are now the one who must secure the data.
• Works offline, for example on a factory floor or in a field office with poor internet.
• Fixed version: the model never changes unless you change it.
• Cost at high volume, once you already own suitable hardware, and the freedom to fine-tune.

Reasons not to:

• Quality: for many hard tasks, the strongest hosted models are still ahead of models you can run on ordinary hardware. Test on your own tasks rather than trusting headlines.
• Hardware: large models need expensive GPUs; small models run on laptops but are slower and weaker.
• Operations: you handle scaling, uptime, security updates and monitoring.

How much memory does a model need?

A model's size is counted in parameters (weights). Each parameter takes memory: 2 bytes at 16-bit precision, 1 byte at 8-bit, and about half a byte at 4-bit. Storing weights at lower precision is called quantisation. It shrinks the model a lot and usually loses a little quality. On top of the weights, running the model needs extra memory for the conversation context. This rough calculator adds 20 percent for that:

BYTES_PER_PARAM = {"16-bit": 2.0, "8-bit": 1.0, "4-bit": 0.5}

def memory_gb(billions, precision, overhead=0.20):
    weights_gb = billions * BYTES_PER_PARAM[precision]     # 1 billion params x 1 byte = about 1 GB
    return weights_gb * (1 + overhead)

for size in (3, 8, 27, 70):
    row = ", ".join(f"{p}: {memory_gb(size, p):5.1f} GB" for p in BYTES_PER_PARAM)
    print(f"{size:>3}B parameters -> {row}")

 3B parameters -> 16-bit:   7.2 GB, 8-bit:   3.6 GB, 4-bit:   1.8 GB
 8B parameters -> 16-bit:  19.2 GB, 8-bit:   9.6 GB, 4-bit:   4.8 GB
27B parameters -> 16-bit:  64.8 GB, 8-bit:  32.4 GB, 4-bit:  16.2 GB
70B parameters -> 16-bit: 168.0 GB, 8-bit:  84.0 GB, 4-bit:  42.0 GB

So a 3 to 8 billion parameter model at 4-bit fits on a typical laptop with 16 GB of memory, while a 70 billion parameter model needs serious hardware even when quantised. The 20 percent overhead is only a rough allowance: long contexts need much more. Treat these numbers as a first estimate and check the model's page for real requirements.

Speed matters too. Local speed is measured in tokens per second, and it depends heavily on your hardware:

reply_tokens = 300                       # about 220 words of English
for name, tps in [("laptop CPU", 8), ("laptop GPU", 35), ("server GPU", 120)]:
    print(f"{name:<11} {tps:>4} tokens/s -> about {reply_tokens / tps:5.1f} s for the reply")

laptop CPU     8 tokens/s -> about  37.5 s for the reply
laptop GPU    35 tokens/s -> about   8.6 s for the reply
server GPU   120 tokens/s -> about   2.5 s for the reply

These speeds are made-up examples for the arithmetic; measure your own. The lesson holds anyway: on slow hardware, ask for short replies.

Running a model with Ollama

Ollama is a popular free tool for running open-weight models on Windows, macOS and Linux. After installing it from its official site, you download a model by name and chat with it in the terminal (model names change; pick one from Ollama's model library):

ollama pull llama3.2
ollama run llama3.2 "Explain GST in two simple sentences."

Ollama also runs a local server with an OpenAI-compatible endpoint at http://localhost:11434/v1/. That means code written for OpenAI's library can talk to your local model by changing the base URL. Ollama's documentation says the API key is required by the library but ignored, so any placeholder works:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")  # placeholder key

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Summarise this complaint in one line: ..."}],
)
print(response.choices[0].message.content)

Other common tools: llama.cpp, a fast engine that runs quantised models in the GGUF file format on ordinary computers, and vLLM, a server designed for serving open models to many users on GPUs.

Mixing local and hosted models

Many organisations use both: a local model for sensitive text, a hosted model for everything else. The decision belongs in code, not in each employee's memory:

import re

SENSITIVE = [r"\b[2-9]\d{3}\s?\d{4}\s?\d{4}\b",          # Aadhaar-like number
             r"\b[A-Z]{5}\d{4}[A-Z]\b",                 # PAN
             r"(?i)\b(salary|diagnosis|medical)\b"]

def choose_backend(text):
    if any(re.search(p, text) for p in SENSITIVE):
        return "local model (data stays on our server)"
    return "hosted model (stronger, pay per token)"

for text in ["Draft a Diwali greeting for our customers in Pune",
             "Summarise Rohan's medical leave note",
             "Check the PAN ABCDE1234F on this form"]:
    print(f"{text[:40]:<40} -> {choose_backend(text)}")

Draft a Diwali greeting for our customer -> hosted model (stronger, pay per token)
Summarise Rohan's medical leave note     -> local model (data stays on our server)
Check the PAN ABCDE1234F on this form    -> local model (data stays on our server)

Because both backends can speak the same API style, the rest of the app does not care which one answered. Run your evaluation set against both, since the local model may need simpler prompts.

Common mistakes

• Assuming "open" means any use is allowed. Read the licence.
• Choosing a model too big for your hardware, then blaming the model for slow replies.
• Expecting a small local model to match the best hosted models without testing.
• Downloading weights from random links, or in unsafe file formats.
• Forgetting that self-hosting makes you responsible for security, updates and monitoring.

Key points

• Open-weight models publish their weights so you can run them yourself; licences differ, so read them.
• Teams self-host for privacy, offline use, fixed versions and cost at volume, at the price of hardware and operations work.
• Memory is roughly parameters times bytes per parameter; quantisation to 8-bit or 4-bit shrinks it a lot.
• Ollama runs models locally with an OpenAI-compatible endpoint at http://localhost:11434/v1/.
• Route sensitive text to local models in code, and evaluate every backend on your own tasks.

BYTES_PER_PARAM = {"16-bit": 2.0, "8-bit": 1.0, "4-bit": 0.5}
SIZES = [1, 3, 8, 14, 27, 32, 70]

def memory_gb(billions, precision, overhead=0.20):
    return billions * BYTES_PER_PARAM[precision] * (1 + overhead)

def largest_model(memory, precision):
    fitting = [s for s in SIZES if memory_gb(s, precision) <= memory]
    return max(fitting) if fitting else None

print("16 GB, 4-bit:", largest_model(16, "4-bit"), "B")
print("16 GB, 8-bit:", largest_model(16, "8-bit"), "B")
print("48 GB, 8-bit:", largest_model(48, "8-bit"), "B")

16 GB, 4-bit: 14 B
16 GB, 8-bit: 8 B
48 GB, 8-bit: 32 B