Design a Real-Time Analytics Pipeline (Food Delivery Example)

The question: design a real-time analytics pipeline for a food delivery platform (think Uber Eats, Swiggy or Zomato). It must track order events and delivery metrics, and serve two audiences: live operational dashboards for restaurant partners (today's earnings, orders delivered so far, orders pending) and daily business reports for executives and finance. This walkthrough follows the data engineering interview framework end to end.

1. Clarify before designing

Resist drawing boxes. Say "I'd like to ask a few clarifying questions first"; the interviewer expects it.

2. Back-of-the-envelope

Events/day:      5M orders × 6 events             = 30M events/day
Average rate:    30M ÷ 86,400 s                   ≈ 350 events/s
Peak rate:       500K orders/h × 6 = 3M events/h ÷ 3,600 s ≈ 833 events/s
→ 350–833 events/s: easy for a small Kafka cluster (3 brokers, 6–12 partitions)

Event size:      one JSON (order_id, event_type, timestamp, restaurant_id, driver_id, customer_id, coordinates…) ≈ 1 KB
Daily volume:    30M × 1 KB                        = 30 GB/day
2-year bronze:   30 GB × 365 × 2                   ≈ 22 TB raw
With Parquet (2–5× compression)                    ≈ 4–11 TB on S3/ADLS: very manageable

Serving layer:   50K active restaurants × ~1 KB of precomputed metrics (10–15 fields)
                 = ~50 MB in Redis: fits in one instance easily

Conclusion: this is a moderate-scale problem. Kafka, storage and the serving store are all comfortably sized, so the design can stay simple. See back-of-the-envelope estimation.

3. Pipeline design: one platform, two consumption paths

Internalize this framing: it's one event stream (the source of truth) with two shaped views of it, each optimized for one consumer's latency and query pattern. That is a Lambda-style design with a streaming path and a batch path.

• Path A, streaming: events land in Kafka partitioned by `restaurant_id`, so each restaurant's events stay in order on one partition. Spark Structured Streaming (or Flink) consumes them, maintains running metrics per restaurant (active orders, orders completed today, earnings so far, average delivery time over the last hour) and writes them to a NoSQL store such as Redis or DynamoDB that the live dashboard reads.
• Path B, batch: the same topic is copied to S3/ADLS as raw Parquet (on AWS, MSK with Firehose delivering to S3). Scheduled Spark jobs orchestrated by Airflow build silver and gold tables, which feed next-morning executive reports.
• The two paths are independent: if the streaming job is down for a deploy, raw events still land in S3, and the batch path can correct anything the live view got wrong.

Likely deep dives: Kafka (at-least-once vs exactly-once, partition count, retention, log compaction, watermarks and late events) or Spark (shuffles, join strategies, adaptive query execution, how you'd lay out the tables). Brush up on whichever you've used. See event streaming with Kafka, delivery semantics, windows and watermarks and Spark architecture.

4. Data model: medallion layers

fact_orders is partitioned or liquid-clustered by order date. See medallion architecture, dimensional modeling and data layout optimization.

5. Data quality, layer by layer

Name the dimension each check belongs to (completeness, accuracy, consistency, freshness, uniqueness), and alert on freshness: the executive report must not silently show yesterday's data. See data quality and data contracts.

6. Scalability and operations

• Backfills: when a metric definition changes (say, cancelled orders stop counting toward totals), rerun the silver and gold jobs for a start and end date, overwriting the affected partitions atomically. See backfills.
• Idempotency: every job can run any number of times on the same input with the same result: MERGE on order_id and partition overwrites instead of blind inserts. Non-negotiable. See idempotency.
• Schema evolution: schema-on-read in bronze (accept new fields as they come), strict schemas in silver and gold, with Delta's merge-schema allowing deliberate additions.
• Failure handling: dead-letter topics for bad events, retries with backoff, consumer-lag and freshness alerts. See pipeline error handling.

What a strong answer shows

Requirements first, numbers that justify each choice, one source of truth with two shaped views, a clear medallion model, quality checks mapped to dimensions, and an operations story (backfills, idempotency, schema changes). It's fine not to cover every topic in depth; showing the structure and narrating each trade-off is what convinces the interviewer.