Unique ID Generation: Snowflake, UUID, ULID and Custom IDs

Every order, message, user and event needs a unique ID. On one database, an auto-increment column does the job. Across many servers and shards, generating IDs that are unique, fast, compact, and ideally sortable by time, without a central bottleneck, is a design problem of its own.

Why auto-increment breaks

A database auto-increment sequence works on one server. Add a second and both happily hand out ID 1 to different users, and the system breaks. Keeping one central sequence instead means every server asks a central authority for each ID: that's a bottleneck, a single point of failure, and a network call added to every write. Tweets, videos and short URLs all need IDs that any server can generate independently, without collisions.

What you usually want from an ID

• Uniqueness across all machines, with no coordination per ID.
• Roughly time-ordered: new IDs sort after old ones, which keeps B-tree inserts at the end of the index (fast) and makes "recent items" queries easy.
• Compact: 64 bits fits in a BIGINT and is cheaper to index than a 128-bit value or a string.
• Not guessable, when IDs are exposed publicly (sequential IDs leak business volume and invite enumeration).

The options

Snowflake IDs

Twitter's Snowflake packs a timestamp, a machine ID and a per-machine sequence into a 64-bit integer, so every server generates IDs locally with no network call:

id = ((now_ms - EPOCH) << 22) | (machine_id << 12) | sequence
# sequence increments within the same millisecond and resets each new millisecond;
# if it overflows (4,096 IDs in one ms), wait for the next millisecond.

• Throughput: about 4 million IDs per second per machine.
• Machine IDs must be unique: assign them from configuration, a coordination service like ZooKeeper or etcd, or derive them from a pod ordinal.
• Clock going backwards (NTP correction) could produce duplicates: refuse to generate (or wait) until the clock passes the last timestamp used.
• IDs are only roughly ordered across machines (within clock skew), which is fine for sorting feeds.

How the bits work, with an example

• 41-bit timestamp: milliseconds since a custom epoch, enough for about 69 years. Different times give different IDs, and IDs sort by creation time.
• 10-bit machine ID: up to 1,024 servers. Twitter's original design split it into 5 bits of data-center ID and 5 bits of worker ID. Each server gets its ID at deployment, so two machines produce different IDs even in the same millisecond.
• 12-bit sequence: a counter within one millisecond on one machine, up to 4,096 IDs per ms; it resets each new millisecond.

Example: at time 1640995200000, server 1 (machine 001) issues an ID ending in sequence 0001; a second request on server 1 in the same millisecond gets 0002. Server 2 (machine 002) also issues sequence 0001 in that millisecond, and its ID still differs because the machine bits differ. When the next millisecond starts, the sequence resets.

import threading
import time

class Snowflake:
    """64-bit IDs: 1 sign bit | 41-bit ms timestamp | 10-bit machine ID | 12-bit sequence."""
    EPOCH = 1577836800000                       # custom epoch: 2020-01-01 UTC, in ms

    def __init__(self, machine_id: int):
        if not 0 <= machine_id < 1024:
            raise ValueError("machine_id must fit in 10 bits (0-1023)")
        self.machine_id = machine_id
        self.sequence = 0
        self.last_ms = -1
        self.lock = threading.Lock()

    def _now(self) -> int:
        return int(time.time() * 1000)

    def next_id(self) -> int:
        with self.lock:
            now = self._now()
            if now < self.last_ms:              # clock moved backwards: refuse, don't duplicate
                raise RuntimeError(f"clock moved back {self.last_ms - now} ms")
            if now == self.last_ms:
                self.sequence = (self.sequence + 1) & 0xFFF      # 12 bits: 0-4095
                if self.sequence == 0:          # 4,096 IDs used this ms: wait for the next
                    while now <= self.last_ms:
                        now = self._now()
            else:
                self.sequence = 0
            self.last_ms = now
            return ((now - self.EPOCH) << 22) | (self.machine_id << 12) | self.sequence

    @classmethod
    def parse(cls, id_: int) -> dict:
        return {"timestamp_ms": (id_ >> 22) + cls.EPOCH,
                "machine_id": (id_ >> 12) & 0x3FF,
                "sequence": id_ & 0xFFF}

server1, server2 = Snowflake(machine_id=1), Snowflake(machine_id=2)
a, b, c = server1.next_id(), server1.next_id(), server2.next_id()
print(a < b, len({a, b, c}) == 3)          # True True: increasing and unique
print(Snowflake.parse(b))                   # {'timestamp_ms': ..., 'machine_id': 1, 'sequence': 0 or 1}
ids = [server1.next_id() for _ in range(100_000)]
print(len(set(ids)) == len(ids), ids == sorted(ids))   # True True

Snowflake trade-offs

• Strengths: no coordination at runtime; roughly 2–3 million IDs per second per machine in practice (4,096 per ms is a theoretical cap of about 4 million/s); time-sortable, so newer tweets have larger IDs; roughly sequential, which keeps B-tree inserts at the right-hand edge of the index.
• Limits: depends on reasonably synchronized clocks, and a clock that jumps backwards could produce duplicates unless the generator refuses or waits (as above); at most 1,024 machines and 4,096 IDs per ms each; machine IDs must be assigned uniquely at setup; and the ID reveals its creation time.

Machine ID coordination is the one setup step. Without it, servers A and B might both pick machine ID 5 and collide. Common strategies:

# 1. Configuration management: assign IDs explicitly
# server-config.yml
server_1: { machine_id: 1 }
server_2: { machine_id: 2 }
server_3: { machine_id: 3 }

# 2. A registry (database or ZooKeeper/etcd) hands out IDs at startup
def get_machine_id(db, hostname):
    existing = db.get(f"machine_id:{hostname}")
    if existing is not None:
        return int(existing)
    new_id = db.incr("next_available_machine_id") - 1      # atomic counter
    if new_id >= 1024:
        raise RuntimeError("out of machine IDs")
    db.set(f"machine_id:{hostname}", new_id)
    return new_id

# 3. Orchestration: a Kubernetes StatefulSet gives pods stable ordinals (app-0, app-1, ...)
#    MACHINE_ID=${HOSTNAME##*-}   ->  0, 1, 2 ...

Unlike UUIDs, which need no agreement at all, Snowflake needs this one-time agreement on who owns which machine ID. Use Snowflake when you need sortable IDs at high throughput (timelines, messages) with fewer than about 1,000 generating servers, and when exposing the creation time is acceptable.

Alternatives in depth

UUID

A UUID v4 is 128 random bits, e.g. f47ac10b-58cc-4372-a567-0e02b2c3d479. Collisions are astronomically unlikely, no coordination or clock is needed, every language supports it, and you can generate IDs offline. The costs: 16 bytes instead of 8, no time ordering, not human-friendly, and poor write performance as a primary key.

Why random keys hurt: sequential IDs (Snowflake, auto-increment) always append at the right edge of the B-tree, where the page is already cached. Random UUIDs land anywhere, so the database must read the right page from disk, split pages when inserting in the middle, and leaves the index fragmented, which wastes cache. In write-heavy systems sequential inserts are often much faster; the exact gap depends on workload, table size and hardware. See database indexing.

Use UUIDs when you can't coordinate machine IDs, for offline-first apps, or when you don't need time ordering. UUID v7 (a timestamp prefix plus randomness) keeps most UUID benefits while being time-ordered.

ULID

A ULID is 128 bits: a 48-bit millisecond timestamp plus 80 random bits, encoded as 26 case-insensitive, URL-safe Base32 characters, e.g. 01F8VYXK67BGC1XPD2YH1W8HTH. It's time-sortable like Snowflake, needs no machine IDs, and its timestamp covers about 8,900 years. It's longer than a Snowflake (26 characters vs up to 19 digits), the 80 random bits allow far more IDs per millisecond than Snowflake's 4,096 per node, and tooling is less mature. Use it when you want sortable IDs without coordinating machine IDs.

Partitioned auto-increment

Give each database server its own start and step: with three servers and step 3, server 1 issues 1, 4, 7, 10, 13; server 2 issues 2, 5, 8, 11, 14; server 3 issues 3, 6, 9, 12, 15.

-- MySQL: server 1 of 3
SET GLOBAL auto_increment_increment = 3;
SET GLOBAL auto_increment_offset = 1;      -- server 2 uses 2, server 3 uses 3

-- PostgreSQL equivalent
CREATE SEQUENCE order_ids START 1 INCREMENT 3;   -- other servers START 2 / START 3

Simple, small (8 bytes), natively supported and ordered per server. But the databases remain bottlenecks and single points of failure, adding or removing servers means renumbering the scheme, and sequential IDs reveal your scale. Fine for smaller systems.

Comparison

*Snowflake needs machine IDs assigned at setup, but no runtime coordination between servers.

A decision framework for interviews

1. Need time-based sorting? Yes → Snowflake, ULID or auto-increment. No → UUID v4.
2. How many servers? Under about 1,000 → Snowflake works; more → ULID or UUID.
3. Can you coordinate machine IDs? Yes → Snowflake; no → ULID or UUID.
4. Must IDs be unguessable? Yes → UUID or other secure random IDs; no → Snowflake or ULID.
5. Write-heavy or read-heavy? Write-heavy → avoid random UUID primary keys (page splits); read-heavy → UUIDs are fine.

In practice: Twitter uses Snowflake for tweets (time ordering at scale); Instagram uses a modified Snowflake with a different bit split; GitHub uses UUIDs for some resources and auto-increment for others; Stripe uses its own prefixed IDs such as cus_1234567890abcdef.

Custom ID formats

Sometimes no standard format fits, and companies design their own:

• Stripe: {prefix}_{random}, e.g. cus_… customers, ch_… charges, sub_… subscriptions, inv_… invoices. You can tell the object type at a glance, and the IDs are URL-safe.
• GitHub: different schemes for different jobs: human-readable repo paths (github.com/user/repo-name), per-repository auto-increment issue numbers (#1, #2, #3), and Git SHA hashes for commits.
• YouTube: 11-character Base64-style IDs like dQw4w9WgXcQ, giving short URLs. They're non-sequential so you can't guess neighbors by incrementing; 11 characters × 6 bits ≈ 66 bits of space (they encode an internal key rather than being fully random), and they're short enough to share in a text message.

Safe characters are a–z, A–Z, 0–9, - and _; characters like + / = & ? # % and spaces need encoding. URL-safe IDs drop straight into paths (/api/orders/ord_456def) with no encoding bugs.

import secrets
import string

class PrefixedIDGenerator:
    """Stripe-style IDs: '<prefix>_<random>' using a cryptographically secure source."""

    def __init__(self, prefix: str, length: int = 16, uppercase: bool = False):
        self.prefix, self.length = prefix, length
        self.alphabet = string.digits + string.ascii_lowercase + (string.ascii_uppercase if uppercase else "")

    def new(self) -> str:
        return f"{self.prefix}_" + "".join(secrets.choice(self.alphabet) for _ in range(self.length))

customers = PrefixedIDGenerator("cus")
charges = PrefixedIDGenerator("ch", length=18, uppercase=True)
print(customers.new())   # e.g. cus_4f0k2m9x7q1a8z3b
print(charges.new())     # e.g. ch_3L4E5Y2eZvKYlo2CqP
# 16 base-36 characters ≈ 82 bits of randomness: collisions are negligible, but still
# insert with a UNIQUE constraint and retry on the (astronomically rare) conflict.

Build custom IDs when users see or share them (branded or human-readable formats), when several object types should be distinguishable, when the API needs stable external IDs separate from internal database keys, for regulatory or audit formats, or to change internal storage later without breaking external APIs. In an interview, start from the standard options (Snowflake, UUID, ULID), explain why they don't fit (say, UUIDs are too long for shareable URLs), then cover the details: cryptographically secure randomness to prevent enumeration, a uniqueness check, and the extra code your team must now maintain.

Short public IDs

For URLs (short links, invite codes), encode the numeric ID in base 62 (a–z, A–Z, 0–9): 7 characters cover 62^7 ≈ 3.5 trillion values. If IDs shouldn't be guessable, don't expose sequential numbers directly: shuffle them with a reversible permutation, or generate random codes and check for collisions. See the URL shortener round.

Choosing

• Single database, internal IDs: auto-increment is fine.
• Distributed system, want simplicity and standard tooling: UUID v7.
• Need compact 64-bit IDs at high volume (messages, events, tweets): Snowflake-style.
• Need short, human-friendly codes: base-62 encoding of a numeric ID or a random code.