Scalability: Vertical vs Horizontal Scaling
What scalability means, vertical vs horizontal scaling, stateless services, and how real systems grow from one server to thousands.
Scalability is a system's ability to handle more load (more users, requests, or data) by adding resources, without being redesigned. A scalable system's cost and latency grow smoothly with load instead of hitting a wall. There are two basic ways to add resources: make a machine bigger, or add more machines.
A story: from one server to many
Say you build an online pet-supplies store for your uncle. It starts like everyone's first web app: a server written in PHP or Node.js, with MySQL or MongoDB behind it, probably on the same machine. The server takes a request, asks the database for data and returns it to the user's device. You rent a $5-a-month VPS (1 GB of memory, a 25 GB SSD) from a provider like DigitalOcean, buy a domain, point its DNS record at the VPS's IP address, and the store is live. Life is simple.
The store gets popular and the little VPS can't keep up. The obvious fix is a bigger server, say $10 a month for 4 GB of memory and an 80 GB SSD. That is vertical scaling.
Business keeps booming: from friends to the whole state, to all 50 states, then to international customers. You keep moving to the next size up, until that stops working:
- One server is a single point of failure. If it goes down, the whole store goes down (and your uncle calls, frantically).
- There's a ceiling, and the price climbs steeply near it. Even very large instances, such as an x1.16xlarge with 64 vCPUs and 976 GiB of memory, or high-memory machines with tens of TiB of RAM, eventually hit limits, and cost grows disproportionately with size.
So you switch to horizontal scaling: instead of one ever-bigger server, run a team of smaller ones, with a load balancer in front acting as foreman: it forwards each request to one of the servers and passes the response back. To make that work, the servers must be stateless: any of them must be able to handle any user's request.
Vertical scaling (scale up)
Vertical scaling means moving to a bigger machine: more CPU cores, more memory, faster disks. It needs no code changes, which makes it the right first move surprisingly often. A single modern server with 64+ cores and hundreds of GB of RAM can serve tens of thousands of requests per second, and a well-tuned PostgreSQL instance on one machine runs many successful businesses.
- Pros: simple, no distributed-systems problems, strong consistency for free, low latency (everything is local).
- Cons: a hard ceiling (the biggest machine you can buy), price grows faster than capacity at the top end, and the machine is a single point of failure. Upgrading usually means downtime.
Horizontal scaling (scale out)
Horizontal scaling means adding more machines and splitting the work between them. It has no practical ceiling and gives you redundancy, since losing one of twenty servers costs 5% of capacity, not 100%. The price is complexity: requests need routing, data needs splitting or copying, and machines can disagree.
The key to scaling out: stateless services
A service is stateless when any instance can handle any request because nothing about a user lives in one instance's memory. Stateless tiers scale out trivially: put a load balancer in front and add instances. To get there, move state somewhere shared:
- Sessions into Redis or a signed token instead of server memory.
- Uploaded files into object storage instead of local disk.
- Background work onto a queue instead of in-process threads that die with the instance.
After that, the hard part of scaling is almost always the data tier, because data can't simply be copied around freely.
Scaling the data tier
| Technique | What it scales | Main cost |
|---|---|---|
| Caching | Reads of hot data | Staleness and invalidation |
| Read replicas | Read throughput | Replication lag: replicas can be slightly behind |
| Sharding | Writes and data size | Cross-shard queries, rebalancing, operational load |
| Denormalization | Read cost per query | Duplicate data to keep in sync |
| Async processing | Spiky writes and slow work | Eventual consistency, more moving parts |
| CDN | Static and cacheable content globally | Invalidation, cost |
How systems typically grow
- One server runs the app and the database. Fine for a prototype.
- Separate the database onto its own machine; scale both vertically.
- Stateless app servers behind a load balancer; sessions in Redis.
- Cache and CDN take most reads off the database.
- Read replicas for the remaining read load; writes still go to one primary.
- Async work via queues for email, image processing, and fan-out.
- Shard the largest tables, or move them to a horizontally scalable store, once writes or data size outgrow one primary.
- Split services and go multi-region for team autonomy and global latency.
Scale for the load you have plus the next order of magnitude, not for a hypothetical billion users. Every step above adds operational cost; take it when an estimate or a metric says you need it. See back-of-the-envelope estimation.
Scalability is not the same as performance
A performance problem means the system is slow for a single user. A scalability problem means it is fast for one user but slow, or failing, under load. Adding servers fixes the second, not the first. A query that takes 2 seconds on an idle database needs an index, not more machines.
Also watch for Amdahl's law: the part of the work that can't be parallelised caps the speedup. If every request takes a lock on one database row, a thousand app servers won't help.
Where it’s used
Famously served one of the busiest sites on the web from a handful of large, vertically scaled SQL Server and web machines, showing how far scaling up can go.
Run thousands of stateless service instances behind load balancers in auto-scaling groups that grow and shrink with traffic.
Scaled PostgreSQL by sharding into thousands of logical shards mapped onto a smaller number of physical servers, moving shards as load grew.
Splits merchants into pods, each with its own database shard, so a flash sale on one store can't take down the others.
In an interview
- Start with a simple design and scale it in response to your own estimates; say which component breaks first and why.
- State that app servers are stateless and where the state lives, then put them behind a load balancer.
- Spend your scaling discussion on the data tier: caching, replicas, then sharding, with the cost of each.
- Separate performance problems from scalability problems: don't add servers to fix a missing index.