Core System Design Challenges and How to Scale a System

Before learning the individual building blocks, it helps to know what problems they exist to solve. Almost every hard part of system design comes from serving a very large number of users by adding more machines. Take away the scale and most design questions shrink back into coding questions. The good news is that the challenges repeat. There are four of them, and once you can name them, the jargon-heavy solutions (sharding, replicas, queues, eventual consistency) make much more sense.

These challenges are also why the non-functional requirements, above all scalability and availability, are the heart of a design interview. Scalability is not a feature you bolt on later: it shapes the whole architecture, which is why systems meant to grow are designed for growing load from the start.

Challenge 1: too many concurrent users

A large user base brings many problems, but the most common and intuitive one is that a single machine or database can only handle so many requests per second (RPS/QPS). Every single-server demo app from a web tutorial works fine until traffic passes that limit; then performance falls off a cliff.

The fix is equally intuitive: repetition. Run several identical copies of the bottlenecked part and spread users across them. When the copies are servers running application logic, this is load balancing. When the copies are data, they are database replicas.

Deeper reading: load balancing and replication.

Challenge 2: too much data to move around

The twin of too many users is too much data. Data becomes "big" when one machine can no longer hold all of it: Google's search index, every tweet ever posted, Netflix's whole film library.

The fix is sharding: split the data into pieces by some logical key so each machine holds only part of it. The key decides which data lives together. If Twitter shards tweets by user_id, all of one user's tweets sit on the same machine, so loading a profile touches one shard.

Deeper reading: sharding and scaling databases.

Challenge 3: the system must be fast and responsive

User-facing apps must feel quick. Aim to respond in under 500 ms; past about 1 second the experience feels broken.

Reads are usually fast once data is replicated and cached: a typical read is a lookup in an in-memory key-value store behind an HTTP endpoint, so for many simple apps the latency is mostly the network round trip.

Writes are where it gets hard. A real write often triggers many queries and updates (save the post, update counters, notify followers, index for search) and can easily take longer than a second. The fix is asynchrony: the server accepts the write, puts it on a queue and responds immediately, and workers do the actual processing in the background. Meanwhile the client has room to keep the experience snappy, for example by showing the new post in the UI straight away and only then loading fresh results. That gap is usually 1–2 seconds, which is enough time for the backend to finish. In practice this is built with a message queue or log such as Kafka.

Deeper reading: message queues and latency vs throughput.

Challenge 4: inconsistent (outdated) state

This challenge is created by the fixes for challenges 1 and 2. Once data is replicated and updated asynchronously, a read can easily hit a copy that hasn't caught up. Inconsistency here almost always means outdated, not random: a user sees an older version, or data that has since been deleted, never garbage.

The solution is mostly at the application level rather than the infrastructure level. Stale reads disappear on their own once replicas and background jobs catch up, so design the product so that briefly seeing old data is acceptable. This is eventual consistency. Most apps live with it happily, especially compared with the alternatives: losing data, or being very slow.

The exceptions are banking and payments, where any inconsistency is unacceptable. Those apps must wait for every step to finish before answering the user, which is why a bank transfer feels much slower than a Google search. See consistency models and the CAP theorem.

The scaling toolkit

Scalability is usually the top non-functional requirement, and a handful of moves cover most designs. Each one targets a different pressure.

Estimate the load first

Before choosing a technique, put numbers on the problem. Estimation shows which quantity actually shapes the design: reads, writes, storage, or the peak burst. Derive the numbers from usage instead of guessing them. The recipe never changes: daily actions ÷ 86,400 seconds gives the average rate; multiply by a peak factor for the burst you must survive. Reads usually dwarf writes, which is why caches and replicas go on the read path.

Round aggressively: these are order-of-magnitude numbers. Here they say the read path needs heavy caching and replicas, writes are modest but need durable storage, and 450 TB means sharding is unavoidable. See back-of-the-envelope estimation and QPS and load.

Decomposition

Break the requirements into services, each owning one business capability (users, orders, payments). Each service has a single responsibility, so it can be scaled, deployed and maintained on its own: the order service can run 20 copies during a sale while the user service stays at 3.

See monolith vs microservices and database per service.

Vertical scaling

Vertical scaling (scaling up) is the brute-force option: move to a bigger machine. Cloud computing made it far more practical. Organizations used to wait weeks for new hardware to be built and shipped; now a larger instance starts in seconds. The ceiling is high, too: AWS offers Amazon EC2 High Memory instances with up to 24 TB of RAM, and Google Cloud has Tau T2D machines tuned for compute-heavy work.

Horizontal scaling

Horizontal scaling (scaling out) runs many identical instances of a service. It works best for stateless services, which keep no user data in memory between requests, because then a load balancer can send any request to any instance. See stateless vs stateful.

Partitioning

Partitioning splits requests and data into shards and spreads them over services or databases, using a logical key such as user ID or region. Requests for users a–h go to instance 1 and its database, i–o to instance 2, and so on.

A simple way to choose the shard is hash(key) mod N. It has a painful failure mode: change the number of shards N and almost every key moves to a different shard. Consistent hashing fixes this by placing both nodes and keys on a ring; each key belongs to the next node clockwise. Add or remove a node and only the keys in one arc move, about 1/N of them.

Deeper reading: sharding and consistent hashing.

Caching

Caching speeds up reads by keeping frequently used data in fast memory, typically Redis or Memcached, so most reads never reach the database. The cache is fast but temporary; the database is slow but permanent.

The hit rate is everything. Database load equals total read load × miss rate, so the last few percent matter most. With 50,000 reads per second:

Going from 90% to 99% cuts database traffic by another 10×. See caching.

Buffering writes with a message queue

Busy systems often have bursts of writes, and the database's disk I/O can't keep up with every write arriving at once. A message queue buffers the writes: the request path only enqueues, and workers drain the queue at a rate the database can sustain. A synchronous operation becomes asynchronous, and the database stays at a safe load instead of crashing.

A worked example: traffic arrives at 220 writes/s but the database handles 100/s.

The queue trades some latency for zero dropped requests. It is not magic, though: if overload is sustained, the backlog grows until even the buffer fills. Watch queue depth and add workers or shed load before that happens. See message queues and backpressure.

Separating reads and writes

Whether a system is read-heavy or write-heavy comes from the business. Social media is read-heavy: people scroll far more than they post. IoT systems are write-heavy: devices report constantly and people rarely look. Because the two sides have such different needs, it pays to handle them separately. There are two levels of separation.

1. Leader–follower replication. All writes go to the leader; followers hold copies and serve reads.

2. CQRS (Command Query Responsibility Segregation). CQRS goes further and uses entirely different data models for the two sides:

• The command (write) side handles creates, updates and deletes, using a model optimized for writes.
• The query (read) side handles reads, using a denormalized model optimized for the queries users run.

Changes flow asynchronously from the command side to the query side, usually as events.

A common real example: MySQL is the source of truth, Elasticsearch serves full-text search and analytical queries, and changes are synced from MySQL to Elasticsearch asynchronously by reading MySQL's binlog with change data capture (CDC). See CQRS and event sourcing.

Combining the techniques

Real systems use several of these at once, each against a different pressure. Decompose the monolith so parts scale independently. Partition and cache to spread load and speed up reads. Separate reads from writes so reads are fast and writes are safe. Then adjust the business logic so bottlenecks are avoided without hurting the user experience.

Changing the business requirements

Sometimes the best scaling move is a product change. It isn't strictly technical, but raising it in an interview shows practical experience and good judgment.

• Spread the peak. Instead of one weekly sale for every product and every user at once, give each category its own day per region: baby products on day 1, electronics on day 2. Traffic becomes predictable, so you can pre-load the cache for tomorrow's category and add read replicas only in the regions that need them.
• Hide the consistency gap. During a busy auction (think eBay's final seconds), show the bidder an immediate "bid placed" message while the backend settles the true order of bids asynchronously. Users see the correct final status once the auction closes.
• Other examples: queue users into a virtual waiting room for concert tickets, cap how often a counter refreshes on screen, or show "1M+" instead of an exact like count.

Where these challenges show up

Every classic design problem is some mix of the four: