Data Engineering Design Patterns: Lambda, Kappa, Mesh, Lakehouse

Data engineering design leans on a small set of recurring architectures. They fall into two families: data processing architectures (how data flows through a system and how events are processed, in real time or in batches) and data management architectures (how data is modeled, stored, owned and governed). Real platforms rarely use just one; most combine two or three.

1. Lambda architecture

Processing splits into a batch layer (complete, accurate, recomputed on a schedule) and a speed layer (fresh but partial), merged at serving time. It gives a good view of history and of right now, and it's fault tolerant: if the real-time path misbehaves, the next batch run corrects the numbers. The costs are two codebases to maintain and the same data flowing through both paths, which makes governance harder.

Example: a streaming service's recommendations. Overnight batch jobs (Spark) compute aggregated insights from all history, while a Kafka + Flink path adjusts recommendations as you watch. Results land in different stores (Cassandra, a relational database, S3) depending on how fresh each needs to be. Full details in Lambda vs Kappa.

2. Kappa architecture

One real-time layer only. "Batch" work is done by replaying events from a durable log such as Kafka through the same streaming code. Lower complexity, easier to maintain and strong real-time support; weaker for heavy batch work, which is slower and compute-intensive when done by replay.

Example: a ride-hailing platform where Kafka is the central nervous system, ingesting GPS pings, rider actions and driver updates so prices and matching react to conditions within seconds. Batch-like results come from windowed aggregations: events between one trigger and the next are combined into a state update, a semi-batch approach on the stream. See stream processing and windows.

3. Event-driven microservices

Each part of an application is its own service, and services communicate through events on a queue rather than direct calls. In an online store, ordering, basket, inventory, payment and shipping are separate services: an OrderPlaced event triggers payment and inventory work without the order service depending on them directly. The queue can be SQS, Kafka or RabbitMQ, and each service can use whatever compute and store suits it (functions plus DynamoDB, containers plus PostgreSQL).

The hard part is processing each message once in effect: with several consumers listening, a message one consumer has already handled must not be processed again by another, wasting compute or double-charging. That needs careful acknowledgement handling plus idempotent consumers. See delivery semantics, idempotency and sync vs async communication.

4. Serverless pipelines

Pipelines built only from managed, event-triggered compute (AWS Lambda, Google Cloud Functions, Azure Functions), orchestrated by a serverless workflow service such as AWS Step Functions. You declare memory and CPU; each event spins up a function, which processes it and shuts down. You pay only for execution time and focus on logic instead of servers.

Example: a smart vending machine. A purchase calls the payment gateway, which calls an API Gateway endpoint that triggers a function to run the transaction logic and record the data; a purchase started from a phone also gets a push notification to confirm through Apple Pay or Google Pay.

Limits: functions have a maximum run time (15 minutes on AWS Lambda), and per-invocation costs add up at sustained high volume. Amazon's Prime Video team famously moved a video-quality monitoring service from a serverless, Step Functions-based design to a single process and reported about a 90% infrastructure cost reduction. Serverless is great for spiky, event-driven work, not automatically for everything. See serverless.

5. CQRS and event sourcing

Common in banking: commands (write-heavy: open account, withdraw funds) and queries (read-heavy: list accounts, check balance) get separate models and pipelines, connected through an event log such as Kafka. With event sourcing, every state change is stored as an immutable event (a complete, auditable history) instead of only the latest balance. Sometimes the write side calls the read side, e.g. after a withdrawal, query the remaining balance. See event sourcing and CQRS.

6. Data mesh

Instead of one central data team and one platform account, each business domain (finance, sales, marketing, HR) owns its data in its own account and publishes it for others to consume without copying it, while a central catalog lists what exists. Ownership and governance are decentralized to the domains. Full details in data mesh.

7. Lakehouse

A data lake is flexible and cheap (any format, object storage) but weak for SQL analytics; a warehouse is fast and structured but more expensive. A lakehouse combines them: open table formats (Delta Lake, Iceberg) on object storage give warehouse-like tables, ACID writes and schema enforcement, queryable by many engines. Full details in lakehouse architecture.

Combining patterns

Each pattern solves a different problem, so real systems mix them: a lakehouse storing data from a Kappa-style stream, domains in a data mesh each running batch pipelines, event-driven services feeding a CQRS read model. In an interview, name the pattern, say what problem it solves in this design, and name its cost.