Vertical vs Horizontal Scaling

Before getting into how scaling is done, it helps to pin down what the word means. For a system, scaling means growing its capacity so it can handle more users and more requests, without the cost or the response time growing just as fast.

When traffic outgrows one server, you have two ways out:

• Vertical scaling (scale up): give the same machine more CPU, more RAM and a faster disk. No code changes.
• Horizontal scaling (scale out): add more machines and put a load balancer in front. Capacity grows roughly with the number of machines, and losing one is survivable.

What that means in practice: scaling up is the fast, cheap fix, but every machine has a top size and it’s still one machine, so when it dies you’re down. Scaling out has no top size and survives failures, but it only works for stateless servers. A session held in one server’s memory breaks it, so that state has to move to Redis, a signed token or the database. Either way, the shared database is usually what stops you next: read replicas come first, sharding much later.

The Supermarket Analogy

#

Scaling up is hiring a faster cashier. It helps, until nobody faster exists. Scaling out is opening more counters. Now someone has to point customers to a free counter, and that’s the load balancer. And every counter still sells from the same stock room. That’s your database.

Vertical scaling

#

The appeal is that nothing changes. Same code, same deploy, same single database connection string. You stop the VM, pick a bigger instance type, and start it again. On most clouds that resize means a restart, so even scaling up costs a few minutes of downtime unless something else takes the traffic meanwhile.

It goes further than people expect. Stack Overflow famously served huge traffic from a handful of big servers for years. Today’s largest cloud instances have hundreds of vCPUs and terabytes of RAM, which is more than most products will ever need for their app tier.

Three things end the road:

1. The ceiling. The biggest box exists, and near the top of the range the price per core climbs because you’re paying for specialist hardware.
2. The software. More cores only help code that uses them. A single-threaded Node.js process or a Python app held back by the GIL won’t go faster on 64 cores than on 8. Neither will a hot lock that every request takes.
3. The single point of failure. One machine means one kernel panic, one bad disk or one botched deploy away from a full outage. A bigger box fails just as completely as a small one.

Horizontal scaling

#

With several servers behind a load balancer, capacity grows roughly with the number of machines. The load balancer spreads requests (round-robin, or least-connections if requests vary a lot in cost), and it runs health checks so a dead server stops getting traffic.

sequenceDiagram
  participant C as Client
  participant LB as Load balancer
  participant S1 as Server 1
  participant S2 as Server 2
  participant S3 as Server 3
  C->>LB: request
  LB->>S1: round-robin
  C->>LB: request
  LB->>S2: round-robin
  LB->>S2: health check
  Note over S2: no answer, marked down
  C->>LB: request
  LB->>S3: skips Server 2
  C->>LB: request
  LB->>S1: skips Server 2

Failover isn’t instant. Between the crash and the failed health checks (a few seconds with typical settings), requests already sent to the dead server fail. Clients and the load balancer need retries for idempotent requests, or users see a burst of errors. This is also how Kubernetes treats replicas: readiness probes play the health-check role, and a Deployment replaces pods that die.

Scaling out is also how you buy safety, not only capacity. Two small servers survive a failure that takes one big server down. The catch is headroom: if three servers each run at 90%, losing one pushes the other two to 135%. They’re over capacity, so one failed server takes the whole fleet down with it. Size the fleet so the remaining servers can carry peak traffic with one gone (N+1). Leave room beyond that too: as Day 1 on latency vs throughput showed, latency climbs steeply well before a server hits 100%. The calculator in Sizing it does that math.

The stateless rule

#

Scale-out only works when any server can answer any request. If a user’s session lives in one server’s memory, the next request can land on a different server and find nothing.

sequenceDiagram
  participant C as Client
  participant LB as Load balancer
  participant S1 as Server 1
  participant S2 as Server 2
  participant R as Redis
  C->>LB: POST /login
  LB->>S1: route
  S1->>S1: session kept in memory
  C->>LB: GET /cart
  LB->>S2: route
  Note over S2: no session here, user looks logged out
  Note over S1,R: fix: sessions live in Redis
  S1->>R: save session
  S2->>R: load session

There are three common fixes, each with a cost:

• Shared session store (Redis, Memcached, or the database). Every request does one extra network hop to load the session. It works with any client.
• Signed token (a JWT or a signed cookie). The client carries the state and any server can verify it with the signing key, so no lookup is needed. Revoking a token before it expires is hard, so keep expiry short.
• Sticky sessions at the load balancer pin a user to one server. It’s a stopgap. Load gets uneven, and when that server dies its users lose their sessions anyway.

Sessions aren’t the only state. Uploaded files on local disk, in-process caches and scheduled jobs that assume one instance all break the same way once there are two servers. Move files to object storage, accept that local caches differ per server, and use a lock or a leader for jobs that must run once.

The database is the wall

#

Each counter sells from the same stock room. Once the app tier is stateless and scaled out, every server talks to one database, and that’s where the next limit shows up. Connections run out first: 20 app servers with a pool of 20 each want 400 connections, and Postgres ships with max_connections = 100. A connection pooler like PgBouncer helps. After that comes CPU and disk on the primary.

Read replicas come first. Most apps read far more than they write, so sending reads to replicas takes load off the primary. The cost is replication lag: a user who just saved a change and reads from a replica might not see it yet. Sharding splits writes across databases by a key, and it comes much later, because cross-shard queries and transactions get hard. Both get their own day in this series. The same idea shows up in Kafka partitions, which are how Kafka spreads writes across brokers.

The same limit follows you into independent services. Each service can scale out on its own, but if they all share one database, that database still caps them all.

Sizing it

#

The model assumes servers fail independently, load splits evenly, capacity adds linearly, and the load balancer and database never fail. Real fleets share racks and deploys, so treat the fleet number as a best case. A dash means the value is undefined, such as the one-down load with a single server.

With the defaults, 2,500 req/s needs 3 servers of 1,000 req/s. Run 4 and each sits at 62.5%, rising to 83.3% with one down. The fleet has enough servers up 99.985% of the time (shown rounded to 99.99), which is about 1.3 hours a year short of capacity. One big server with the same 99.5% uptime is down about 43.8 hours a year. Set servers to 3 and the one-down load goes to 125%: three servers carry the peak, but they can’t absorb a failure.

In practice

#

Start by scaling up. It’s the cheapest fix and buys you time while you make the app stateless. Then scale out, for capacity and for safety, and expect the database to be the next wall. The back-of-the-envelope estimation note covers how to get the peak request rate you’d put in the calculator. Scale up to buy time. Scale out to survive.