Delivery Guarantees: At-Most-Once, At-Least-Once and Exactly-Once

Networks drop messages and machines crash at inconvenient moments, so messaging systems can't promise both "never lose a message" and "never deliver it twice" without help. Delivery semantics describe which failure you accept: at-most-once (may lose), at-least-once (may duplicate) or exactly-once (neither, at a cost and usually only within a closed system).

Where duplicates and losses come from

The core problem: "do the work" and "record that it's done" are two separate steps. Whichever order you do them in, a crash between them causes either a loss or a duplicate.

The three guarantees

Each guarantee, step by step

A consumer reads an event, starts processing, and crashes. Is the event lost, or processed again? It depends on when the consumer acknowledges (commits its offset) relative to doing the work.

At-most-once: fire and forget

Each event is processed zero or one time. A logging pipeline commits event X, crashes while writing to the database, restarts after X, and X is gone. Use it when occasional loss is fine and speed matters most: metrics where a few missing points don't change trends, log aggregation with tolerable gaps. Simplest and lowest latency, but loses data on failure.

At-least-once: never lose, may duplicate

Each event is processed one or more times: nothing is lost, but duplicates happen, so downstream must cope:

• Idempotent operations: an upsert by key gives the same result run once or twice; a plain INSERT duplicates.
• Deduplication by event ID: store each event's unique ID and skip IDs already seen.
• Naturally idempotent changes: "set balance to $100" is idempotent; "add $10 to balance" is not.

Most production stream processing uses at-least-once: no data loss, slightly more latency than at-most-once, and duplicates handled in your logic. See idempotency.

Exactly-once: no loss, no duplicates

The processing result and the input acknowledgment commit together atomically: both happen or neither. Flink and Kafka Streams do this with distributed snapshots (checkpoints): periodically capture what's been consumed, what's in flight and what's been output. On failure, restore the last checkpoint and resume; events after it are reprocessed, but their outputs sit in a transactional sink and commit only when the next checkpoint completes, so nothing is emitted twice.

The catch: this guarantee usually holds inside the stream processing system. Writing to an external database or API exactly once end to end needs a sink whose transactions coordinate with checkpoints, idempotent writes to the sink, or a way to roll back external writes; otherwise you get exactly-once processing but at-least-once delivery to the sink. Use it where correctness is critical (billing, financial transactions, inventory), accepting checkpoint latency and more complexity.

Many production systems choose at-least-once plus idempotent operations, which gives "effectively-once" results with less complexity: upserts instead of inserts, unique event IDs deduplicated at the sink, and naturally idempotent operations. That pushes the complexity into application logic you control rather than distributed transaction protocols that are hard to get right.

Exactly-once is really "effectively once"

Delivering a message exactly once over an unreliable network is impossible in general. What systems actually provide is exactly-once processing effects: the message may arrive twice, but its effect is applied once. Two techniques get there:

• Idempotent processing: record each message ID (or a business key) with the result, in the same transaction as the side effect, and skip IDs already seen. See idempotency.
• Atomic commit of output and progress: store the result and the consumer's position together, so they can't diverge. Kafka transactions do this for Kafka-to-Kafka pipelines (consume, produce, and commit offsets atomically); a database consumer can store its offset in the same table transaction as its writes.

BEGIN;
  INSERT INTO processed_messages (message_id) VALUES ('msg-81');   -- fails if seen before
  UPDATE accounts SET balance = balance - 500 WHERE id = 'acct-9';
COMMIT;
-- A redelivered msg-81 hits the unique constraint, the transaction rolls back,
-- and the consumer simply acknowledges it.

Exactly-once inside Kafka doesn't extend to side effects outside it. Sending an email or calling a payment API from a Kafka transaction can still happen twice; those calls need their own idempotency keys.

Producers matter too

Duplicates also appear when a producer retries a publish because the acknowledgement was lost. Kafka's idempotent producer assigns sequence numbers so the broker discards duplicate writes; elsewhere, include a unique event ID so consumers can deduplicate.

Choosing

• Default to at-least-once delivery with idempotent consumers. It's simple, robust and works with every broker.
• Use at-most-once only where losing data is genuinely fine and duplicates are worse or pointless.
• Reach for transactional exactly-once in stream processing where counts and sums must be exact and the whole pipeline is inside one system.