Design a URL Shortener (bit.ly): Step-by-Step System Design

A URL shortener turns a long URL into a short, unique alias that redirects to the original: bit.ly, TinyURL, Twitter's t.co. The alias is a short fixed-length string. The service must handle millions of URLs, keep every short URL unique and permanent, redirect in near real time under heavy traffic, and often track link usage such as click counts and locations. This walkthrough follows the system design template and the problem decomposition framework.

1. Functional requirements

Extract the verbs from the problem statement; each becomes an operation:

Out of scope for this version: user accounts, custom aliases, link editing and expiry (each is a short extension at the end).

Scale

• 100M daily active users
• Read:write ratio 100:1
• 1 million new URLs per day
• About 500 bytes per entry
• Keep data for 5 years

Back-of-the-envelope

Writes are modest; reads dominate, so the design centers on fast, cached redirects. See back-of-the-envelope estimation and measuring load.

2. Non-functional requirements

Extract the adjectives and descriptive phrases:

3. Data model

From the nouns: "URL" and "alias" become a URLMapping; "link usage" and "click counts" become Analytics; "persistent" adds a created_at. Ownership is split so each scales independently: the URL Shortening Service owns URLMapping (and guarantees unique IDs); the Analytics Service owns Analytics (absorbing heavy counter writes without slowing redirects).

4. API

302 vs 301: a 301 (permanent) lets browsers cache the redirect and skip your servers next time, which saves load but loses click counts; a 302 (temporary) brings every click through you, so analytics stay accurate. See API design.

5. High-level design

Shortening

The service generates a unique short alias (from a unique integer ID, below), stores the long→short mapping in a highly available database (DynamoDB or Cassandra fit: simple key lookups, easy horizontal scale), handles errors, and guarantees each alias is unique across all users.

Redirection

Reads are 100× writes, so redirection is optimized for throughput and latency. An API gateway now routes POSTs to the shortening service and GETs to the redirect handler, which looks the code up through a read-through cache (Redis with a cache library: on a miss the cache itself loads from the database and keeps the result), then answers 302 with the original URL in Location. See cache strategies.

Analytics

The gateway also sends each GET to an Analytics Service that increments a counter per short URL in an in-memory store (Redis INCR) and periodically flushes the counters to the database, keeping counting fast and off the redirect path. A more robust variant publishes a click event to Kafka and aggregates it in a stream job (see real-time analytics).

6. Deep dives

What properties must the IDs have?

Two: global uniqueness (two long URLs must never share a short URL) and shortness (production shorteners use about 5–8 characters, e.g. shorturl.at/xLMPr, t.ly/ecgGp, tinyurl.com/e9enh3uz). The approach: generate a unique integer ID, then encode it into a short readable string.

How do we generate unique IDs?

("Integer" means a whole number, whatever base we write it in: 123456 decimal is 0x1e240 in hex and 0b11110001001000000 in binary.)

With 62 possible prefixes and 6 base62 characters of sequence, each generator can issue 56.8 billion codes, and collisions are impossible because prefixes differ.

How do we encode the IDs?

ALPHABET = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"   # 62 URL-safe chars

def encode(n: int) -> str:
    if n == 0:
        return ALPHABET[0]
    out = []
    while n:
        n, rem = divmod(n, 62)
        out.append(ALPHABET[rem])
    return "".join(reversed(out))

def decode(s: str) -> int:
    n = 0
    for ch in s:
        n = n * 62 + ALPHABET.index(ch)
    return n

class IdGenerator:
    """One per machine: a 1-character prefix (the machine/shard ID) + a local sequence."""

    def __init__(self, prefix: str, start: int = 0):
        self.prefix, self.next_seq = prefix, start   # persist next_seq (or reserve ranges) in production

    def new_short_code(self, width: int = 6) -> str:
        seq, self.next_seq = self.next_seq, self.next_seq + 1
        return self.prefix + encode(seq).rjust(width, "0")

print(encode(123456789), decode(encode(123456789)))   # 8m0Kx 123456789
gen = IdGenerator(prefix="a", start=1_000_000)
print(gen.new_short_code(), gen.new_short_code())      # a004c92 a004c93
print(62 ** 6, 62 ** 7)                                # 56800235584 3521614606208

How do we scale for high traffic?

Request handlers are stateless HTTP servers, so they scale by adding instances behind a load balancer. The ID generators need more thought. Use the machine ID prefix as the shard key: shard the database and the ID generators by the same prefix, so each generator writes to exactly one database shard. A common pattern: every write path is independent.

• Scalability: a new machine gets a new prefix and its own shard, with no change to existing ones.
• Concurrency: independent write paths write in parallel without conflicts.
• Isolation: a failing generator or shard doesn't affect the others.
• Simpler data management: backups, indexing and maintenance happen per shard.

Reads use the same prefix: on a cache miss for a82c7w, the handler goes straight to shard a. The cache can be sharded by prefix too if it grows. See sharding.

Scaling request handlers and ID generators independently

Run ID generation as its own small service rather than inside every handler. Handlers scale with redirect traffic (hundreds of instances); generators scale with write traffic (a handful), and their number is bounded by the prefixes you've assigned. Each generator persists its counter, or better, reserves ranges: take the next 1,000 sequence numbers in one durable write and hand them out from memory; a crash just skips the unused part of a range, which is harmless.

What about pre-generating IDs in bulk?

A key generation service can pre-compute codes into a "free keys" table; handlers fetch batches (say 1,000 at a time) into memory and hand them out with no per-request coordination. It moves ID work off the request path and makes code selection O(1). Costs: storage for unused keys, marking keys as used atomically so two servers never take the same batch, and losing in-memory keys when a server dies (fine: there are trillions). In practice, range reservation gives the same benefit with less storage.

Other extensions

• Same long URL twice? Either always create a new code (simple, lets each user track their own link) or look up an index on original_url (or its hash) to reuse the existing code.
• Custom aliases: check-and-insert with a conditional write (attribute_not_exists / unique constraint), and keep custom aliases out of the generated prefix space.
• Expiry: store expires_at, return 410 Gone after it, and let a TTL or cleanup job delete old rows.
• Security: scan submitted URLs against malware/phishing lists, rate-limit creation per user and IP, and never allow redirects to javascript: URLs. See rate limiting.
• Hot links: a viral link is served from cache (and a CDN edge if you cache redirects there), so the database never sees its spike. See hot partitions.