Working with Web APIs

Weather apps, payment gateways, train-status sites and maps all share data through web APIs. An API (application programming interface) is a set of URLs that a program, not a person, can call to ask for data or to make something happen. Most modern APIs speak HTTP, the same protocol as web pages, and send data back as JSON.

In this lesson you will learn how an HTTP request and response are built, call APIs with Python's built-in urllib.request and with the popular requests package, handle timeouts and errors properly, keep API keys safe, and meet httpx, a modern alternative.

Requests and responses

Every API call is a request from your program and a response from the server.

A request has a method (what you want to do), a URL (what you want to do it to), headers (extra information such as your API key or the data format) and sometimes a body (data you are sending). The response has a status code, its own headers, and a body, usually JSON.

Status codes are three-digit numbers grouped by their first digit: 2xx means success, 4xx means your request had a problem, and 5xx means the server failed. Python's http.HTTPStatus knows all of them:

from http import HTTPStatus

for code in (200, 201, 204, 400, 401, 404, 429, 500, 503):
    status = HTTPStatus(code)
    side = "client error" if status.is_client_error else "server error" if status.is_server_error else "success"
    print(code, status.phrase, "-", side)

200 OK - success
201 Created - success
204 No Content - success
400 Bad Request - client error
401 Unauthorized - client error
404 Not Found - client error
429 Too Many Requests - client error
500 Internal Server Error - server error
503 Service Unavailable - server error

429 Too Many Requests deserves a mention: many APIs limit how many calls you can make per minute, and send 429 when you go over. Slow down and try again later.

The standard library: urllib.request

Python can call APIs without installing anything. urllib.request.urlopen() sends a GET request and returns a response you can read inside a with block. Always give a timeout in seconds, so a dead server cannot freeze your program.

import json
import urllib.request

url = "https://jsonplaceholder.typicode.com/users/3"
request = urllib.request.Request(url, headers={"User-Agent": "college-project/1.0"})

with urllib.request.urlopen(request, timeout=10) as response:
    print(response.status, response.headers["Content-Type"])
    user = json.loads(response.read().decode("utf-8"))

print(user["name"], "|", user["email"], "|", user["address"]["city"])

200 application/json; charset=utf-8
Clementine Bauch | Nathan@yesenia.net | McKenziehaven

response.read() returns raw bytes, so you decode them to text and parse the JSON with json.loads() (see JSON & CSV). A Request object lets you add headers; some servers reject requests that do not send a User-Agent.

urlopen() raises urllib.error.HTTPError for 4xx and 5xx answers, and urllib.error.URLError when there is no answer at all, for example when the address is wrong or the internet is down. Catch HTTPError first, because it is a kind of URLError.

import json
import urllib.error
import urllib.request

def get_json(url):
    try:
        with urllib.request.urlopen(url, timeout=10) as response:
            return json.load(response)
    except urllib.error.HTTPError as error:      # the server answered with 4xx/5xx
        print("HTTP error:", error.code, error.reason)
    except urllib.error.URLError as error:       # no answer at all
        print("Network problem:", error.reason)
    return None

print(get_json("https://jsonplaceholder.typicode.com/users/999"))
print(get_json("https://no-such-host.example/data"))

HTTP error: 404 Not Found
None
Network problem: [Errno 11001] getaddrinfo failed
None

The exact network error text depends on your operating system; this one is from Windows. urllib is fine for simple GET requests, but sending data, handling sessions and reading errors take more code than with requests.

The requests library

requests is the most widely used HTTP package for Python. It is not part of the standard library, so install it inside a virtual environment:

python -m pip install requests

requests.get() takes the URL, an optional params dictionary that becomes the query string after ?, and a timeout. The response has .status_code, .headers, .text and a .json() method that parses the body for you.

import requests

response = requests.get(
    "https://jsonplaceholder.typicode.com/posts",
    params={"userId": 2},
    timeout=10,
)
print(response.status_code, response.url)

posts = response.json()
print(len(posts), "posts")
print(posts[0]["title"])

200 https://jsonplaceholder.typicode.com/posts?userId=2
10 posts
et ea vero quia laudantium autem

Notice how params built ?userId=2 for you, including any escaping that special characters need. To send data, use requests.post() with json=. It converts your dictionary to JSON and sets the right Content-Type header.

import requests

new_post = {"title": "Rohtak college fest", "body": "Fest starts on Friday", "userId": 7}
response = requests.post("https://jsonplaceholder.typicode.com/posts", json=new_post, timeout=10)
print(response.status_code, response.reason)
print(response.json())

201 Created
{'title': 'Rohtak college fest', 'body': 'Fest starts on Friday', 'userId': 7, 'id': 101}

The server answered 201 Created and sent back the new post with its id. (JSONPlaceholder only pretends to save it.) When you make many calls to the same server, create a requests.Session() and call session.get(...): it reuses the connection, which is faster, and keeps headers and cookies between calls.

Timeouts and errors

Network code fails in more ways than normal code: the server can be slow, down, or return an error page. Handle each case on purpose.

A status like 404 or 500 is not an exception in requests; the call succeeds and you get a response with that code. Call raise_for_status() to turn 4xx and 5xx codes into a requests.HTTPError.

import requests

def fetch_user(user_id):
    url = f"https://jsonplaceholder.typicode.com/users/{user_id}"
    try:
        response = requests.get(url, timeout=(3, 10))
        response.raise_for_status()               # 4xx or 5xx becomes an exception
    except requests.Timeout:
        print("The server took too long to answer")
    except requests.HTTPError as error:
        print("Server said:", error.response.status_code, error.response.reason)
    except requests.ConnectionError:
        print("Could not connect; check the internet connection")
    else:
        return response.json()
    return None

user = fetch_user(5)
print(user["name"] if user else "No user")
print(fetch_user(500))

Chelsey Dietrich
Server said: 404 Not Found
None

All of these exceptions inherit from requests.RequestException, so you can catch that one class when you do not need different messages.

Once you have the JSON, treat it like any data from outside: check it before you trust it. Fields may be missing or empty. This part is runnable, using a saved response body:

import json

body = '''{"city": "Pune", "forecast": [
    {"day": "Mon", "max_c": 31, "rain_mm": 0},
    {"day": "Tue", "max_c": 29, "rain_mm": 12.5},
    {"day": "Wed", "max_c": null}
]}'''

data = json.loads(body)
for day in data.get("forecast", []):
    max_c = day.get("max_c")
    rain = day.get("rain_mm", 0)
    temp = f"{max_c}°C" if max_c is not None else "no data"
    print(f"{data['city']} {day['day']}: {temp}, rain {rain} mm")

Pune Mon: 31°C, rain 0 mm
Pune Tue: 29°C, rain 12.5 mm
Pune Wed: no data, rain 0 mm

JSON null becomes Python None, and .get() with a default copes with fields the server left out.

API keys

Most real APIs need an API key, a secret string that identifies your account. Usually you send it in a header, often as Authorization: Bearer <key>. Never write the key into your code, because code ends up on GitHub and in screenshots. Read it from an environment variable instead:

import os
import requests

api_key = os.environ["WEATHER_API_KEY"]          # set outside the program
response = requests.get(
    "https://api.example.com/v1/forecast",
    params={"city": "Kochi"},
    headers={"Authorization": f"Bearer {api_key}"},
    timeout=10,
)

This snippet shows the pattern only; api.example.com is a placeholder, so use the URL and header your API's documentation gives.

httpx: a modern alternative

httpx offers almost the same interface as requests, plus async support for use with asyncio and optional HTTP/2. It also applies a 5-second timeout by default. Install it with python -m pip install httpx.

import httpx

with httpx.Client(base_url="https://jsonplaceholder.typicode.com", timeout=10) as client:
    response = client.get("/todos", params={"userId": 1, "completed": "true"})
    response.raise_for_status()
    todos = response.json()

print(response.status_code, response.http_version)
print(len(todos), "finished tasks, first:", todos[0]["title"])

200 HTTP/1.1
11 finished tasks, first: et porro tempora

A Client works like a requests Session, and base_url saves repeating the start of every URL. For async code, httpx.AsyncClient is used with async with and await client.get(...).

Common mistakes

• No timeout. A hung server freezes the program. Always pass timeout=.
• Assuming success. A 404 or 500 response is not an exception in requests; call raise_for_status() or check status_code.
• Hard-coding API keys. Use environment variables and keep keys out of Git.
• Building query strings by hand. Pass params= so values are escaped correctly.
• Calling an API in a tight loop. You will hit rate limits (429). Cache results and space out calls.

Key points

• An HTTP request has a method, URL, headers and an optional body; the response has a status code, headers and a body.
• 2xx means success, 4xx means a problem with the request, 5xx means a server problem.
• urllib.request.urlopen() works without installing anything; requests is simpler for real work.
• Use params= for query strings, json= to send JSON and .json() to read it.
• Always set a timeout and handle Timeout, HTTPError and ConnectionError.
• Keep API keys in environment variables; httpx adds async support and default timeouts.

import requests

def todo_summary(user_id):
    try:
        response = requests.get(
            "https://jsonplaceholder.typicode.com/todos",
            params={"userId": user_id},
            timeout=10,
        )
        response.raise_for_status()
    except requests.RequestException as error:
        print("Request failed:", error)
        return None
    todos = response.json()
    done = sum(1 for todo in todos if todo["completed"])
    return {"total": len(todos), "completed": done}

print(todo_summary(1))

{'total': 20, 'completed': 11}