Vertical & Horizontal Scaling

By the end of this lesson you will be able to choose between scaling up and scaling out, make an app server stateless so it can be copied freely, and reason about auto-scaling for daily traffic peaks.

Two ways to grow

Scalability is the ability of a system to handle more load by adding resources. There are two ways to add them.

Vertical scaling (scaling up) means using a bigger machine: more CPU cores, more RAM, faster disks. Your code does not change.

Horizontal scaling (scaling out) means adding more machines and spreading the work across them.

Think of a sweet shop in Jaipur before Diwali. Vertical scaling is buying a bigger kadhai and a faster stove for the one cook. Horizontal scaling is hiring four more cooks with their own stoves. The bigger kadhai is easy, but there is a limit to how big a kadhai can be. More cooks can go on growing, but now someone must divide orders between them and make sure they all follow the same recipe.

A sensible path for most products: start simple on one machine, scale up while it is cheap, and scale out the stateless parts as soon as you need redundancy or more capacity.

Stateless servers are easy to copy

A server is stateless if it keeps no user data in its own memory between requests. Every request carries what is needed (such as a token), and anything that must be remembered is stored in a shared place, such as a database or a cache.

A stateful server keeps something important locally, such as a user's shopping cart in a Python dictionary. That breaks the moment there are two servers: the next request may land on the other server, which has never heard of the cart.

Step 1: run the model below. Three servers keep carts in their own memory, and requests are spread among them.

class AppServer:
    def __init__(self, name, session_store=None):
        self.name = name
        self.local = {}                       # memory inside this one server
        self.store = session_store            # shared store, if any

    def _sessions(self):
        return self.store if self.store is not None else self.local

    def add_to_cart(self, user, item):
        self._sessions().setdefault(user, []).append(item)

    def view_cart(self, user):
        return self._sessions().get(user, [])

def run(servers):
    requests = [("add", "kurta"), ("add", "dupatta"), ("view", None)]
    for i, (action, item) in enumerate(requests):
        server = servers[i % len(servers)]    # load balancer sends each request elsewhere
        if action == "add":
            server.add_to_cart("priya", item)
        else:
            print(f"  {server.name} shows cart: {server.view_cart('priya')}")

print("Stateful servers (carts in local memory):")
run([AppServer("app-1"), AppServer("app-2"), AppServer("app-3")])

shared = {}                                    # stands in for Redis or a database
print("Stateless servers (carts in a shared store):")
run([AppServer("app-1", shared), AppServer("app-2", shared), AppServer("app-3", shared)])

Stateful servers (carts in local memory):
  app-3 shows cart: []
Stateless servers (carts in a shared store):
  app-3 shows cart: ['kurta', 'dupatta']

Step 2: read the result. With local memory, Priya's cart looks empty because her items went to app-1 and app-2. With a shared store, any server can answer. Stateless servers can be added, removed or restarted at any time, which is what makes horizontal scaling and auto-scaling possible.

The usual shape of a scaled-out web app

                +----------------+
users  ------>  | Load balancer  |
                +-------+--------+
         +--------------+--------------+
         v              v              v
    +---------+    +---------+    +---------+
    |  app-1  |    |  app-2  |    |  app-3  |    stateless, identical copies
    +----+----+    +----+----+    +----+----+
         +--------------+--------------+
                v                v
         +------------+   +-------------+
         | Redis      |   | Database    |    shared state lives here
         | (sessions) |   | (orders)    |
         +------------+   +-------------+

Notice where the hard part moved. App servers are now easy to multiply. The database is shared by all of them, so it becomes the next bottleneck. Most of the "Data at Scale" section of this course, starting with Database Replication, is about scaling that last box.

Auto-scaling: following the daily curve

Traffic is not flat. A food delivery app is quiet at 4 am and busy at 1 pm and 9 pm. Auto-scaling adds servers when load rises and removes them when it falls, so you do not pay for peak capacity all day.

A common rule is to keep average CPU usage near a target, such as 60%, leaving headroom for sudden spikes and for new servers to start up.

Step 1: model one day of traffic and compute how many servers the rule asks for each hour.

import math

capacity_per_server = 400        # requests/second one server handles at 100% CPU
target_utilisation = 0.60
min_servers, max_servers = 2, 40

# requests/second at each hour of the day for a food app in Pune
hourly = [300, 200, 150, 120, 120, 200, 600, 1200, 1800, 2000, 2600, 4800,
          7200, 5200, 2600, 2000, 2200, 3000, 4200, 6400, 8000, 6000, 2500, 900]

total_server_hours = 0
for hour, rps in enumerate(hourly):
    want = math.ceil(rps / (capacity_per_server * target_utilisation))
    servers = min(max(want, min_servers), max_servers)
    total_server_hours += servers
    if hour % 3 == 0 or rps >= 7000:
        print(f"{hour:02d}:00  {rps:5} req/s -> {servers:2} servers")

fixed = max(math.ceil(max(hourly) / (capacity_per_server * target_utilisation)), min_servers) * 24
print(f"Auto-scaled: {total_server_hours} server-hours; fixed for the peak: {fixed}")

00:00    300 req/s ->  2 servers
03:00    120 req/s ->  2 servers
06:00    600 req/s ->  3 servers
09:00   2000 req/s ->  9 servers
12:00   7200 req/s -> 30 servers
15:00   2000 req/s ->  9 servers
18:00   4200 req/s -> 18 servers
20:00   8000 req/s -> 34 servers
21:00   6000 req/s -> 25 servers
Auto-scaled: 282 server-hours; fixed for the peak: 816

Step 2: compare the last line. Auto-scaling uses far fewer server-hours than running peak capacity all day. The minimum of 2 servers keeps redundancy even at 3 am, because one server is never enough for safety.

In practice the rule lives in configuration. Here is how a Kubernetes HorizontalPodAutoscaler expresses "keep CPU near 60%, between 2 and 40 copies":

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 2
  maxReplicas: 40
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

Common mistakes

• Keeping state in app server memory, then adding a second server and losing carts and logins.
• Running exactly one copy of anything important. Two is the minimum for availability.
• Scaling app servers when the database is the bottleneck. More app servers then just send more load to the struggling database.
• Relying on reactive auto-scaling for sudden, scheduled spikes. Pre-scale for known events.

Key points

• Vertical scaling means a bigger machine; horizontal scaling means more machines.
• Vertical is simple but limited and has a single point of failure; horizontal grows further and survives failures.
• Stateless app servers keep shared data in a database or cache, so any copy can serve any request.
• Once app servers scale out, the shared database becomes the next bottleneck.
• Auto-scaling follows traffic with a utilisation target, but pre-scale for known spikes.

import math

hourly = [50, 80, 3000, 9000, 4000, 600]
per_server, target, minimum = 500, 0.5, 2
total = 0
for rps in hourly:
    n = max(math.ceil(rps / (per_server * target)), minimum)
    total += n
    print(rps, "->", n)
peak = max(math.ceil(max(hourly) / (per_server * target)), minimum)
print("auto-scaled:", total, "server-hours; fixed:", peak * len(hourly))

50 -> 2
80 -> 2
3000 -> 12
9000 -> 36
4000 -> 16
600 -> 3
auto-scaled: 71 server-hours; fixed: 216