Data Engineering Architecture: From Sources to Trusted Data Products

A data engineering platform turns records from operational systems, files, and event streams into dependable datasets that analysts, products, and models can use. A useful design is not just a collection of pipelines: it defines how data is captured, replayed, validated, governed, and served, with a clear owner for each step.

A common end-to-end shape

Keep the raw layer close to what the source sent. It is the recovery point for reprocessing after a bug, a new business rule, or a changed schema. Curated tables should be easier to query, documented, and owned by a team. Separate storage formats and compute engines are fine; the boundaries and contracts matter more than a specific vendor.

Choose ingestion by source and freshness

The freshness target should drive the architecture. If a report is used once a day, a scheduled batch is usually easier to operate than a permanent streaming job. If a risk decision must be made during checkout, a seconds-late batch is not enough. Many platforms combine both patterns; see batch vs stream processing and event streaming.

Make pipelines recoverable

• Idempotency: write each logical partition or batch so a retry replaces or safely merges its output instead of double-counting it.
• Checkpoints and offsets: persist progress only after output is durable; on restart, expect to replay some input and make consumers tolerate duplicates.
• Backfills: keep raw data long enough to rebuild a derived table, and expose date-range or versioned reprocessing rather than ad-hoc scripts.
• Schema evolution: add compatible fields, version breaking changes, and reject or quarantine malformed records instead of silently dropping them.
• Late and out-of-order events: use event time, watermarks, and an explicit correction policy for streaming aggregates.
• Orchestration: model dependencies, retries, timeouts, and concurrency limits; alert on missed service-level objectives, not only failed processes.

A green job status does not prove that the data is useful. Track freshness, completeness, duplicates, schema changes, and business-level assertions alongside pipeline health.

Quality, governance and serving

Treat datasets as products with owners, descriptions, sensitivity labels, and supported consumers. Run checks at ingestion (shape and required fields), after transformations (uniqueness and referential integrity), and at the business boundary (for example, totals reconcile with the source). Keep lineage from source through each transformation so an incident can identify affected reports and features.

• A columnar lake format is a good fit for inexpensive retained history and large scans; a warehouse is tuned for governed analytical queries.
• A low-latency application API may need a separate serving table or key-value store; do not force user-facing traffic onto a warehouse.
• Partition by common filter dimensions such as date, but avoid tiny files and very high-cardinality partitions.
• Use least-privilege access, encryption, retention limits, and deletion propagation for personal or regulated data.
• Measure freshness, pipeline duration, failure rate, bytes scanned, and the cost per useful dataset or query.

A practical interview sequence

1. Clarify users, source systems, volume, freshness, retention, and correctness requirements.
2. Draw source ingestion into a durable raw layer before adding transformations.
3. Choose batch, streaming, or both from the freshness requirement and explain late-data handling.
4. Name the curated datasets and the serving path for each consumer.
5. Close with replay, idempotency, data quality, access controls, and operational metrics.