Sign inStart practicing
Start practicing Sign in

Data Engineering Roadmap 2026: From Pipelines to Interviews

A step-by-step data engineering roadmap for 2026: Spark batch, Kafka streaming, data modeling, lakehouse storage, data quality, backfills and interviews.

Beginner 3 min read · Updated

This data engineering roadmap takes you from the shape of a modern data platform to the details interviewers probe: how data is ingested with change data capture and event streams, modelled into fact and dimension tables, processed in batch with Spark and in real time with stream processors, orchestrated and backfilled safely, and checked for quality before anyone trusts a dashboard. Each step links to a free guide and a design round that tests it.

Data engineering interviews usually include SQL, a coding round and a pipeline or platform design round. This roadmap covers the design round, and the modelling steps help with the SQL one.

Builds on the core system design roadmap. Do that first, or skip the parts you already know; this one only covers what data engineering interviews go deeper on.

Tick topics off as you read them

  1. Step 1The data engineering design round

    How the round differs from backend design, and the six steps every answer follows.

  2. Step 2The shape of a data platform

    Sources, layers, consumers and where analytics data lives.

  3. Step 3Batch processing

    Large, repeatable transformations on a schedule.

    Practice:Reusable ETL/ELT frameworkData engineering · AdvancedPayments reconciliation pipelineData engineering · Intermediate

  4. Step 4Streaming and ingestion

    Moving events and database changes as they happen.

    Practice:Change data capture pipelineData engineering · Intermediate · FreeClickstream event tracking at scaleData engineering · AdvancedCentralized log ingestion pipelineData engineering · IntermediateIoT sensor telemetry platformData engineering · Advanced

  5. Step 5Architecture patterns

    The recurring designs and when to combine them.

    Practice:Ad click aggregatorData engineering · AdvancedVideo playback analytics with sessionsData engineering · Advanced

  6. Step 6Data modeling

    Designing tables analysts can trust, and defending the choices.

    Practice:Data warehouse for an online storeData engineering · Beginner · FreeCustomer warehouse with SCD Type 2Data engineering · Intermediate · Free

  7. Step 7Storage and file formats

    Where data lives, in which format, laid out so queries stay cheap.

    Practice:Lakehouse for analytics and MLData engineering · Advanced

  8. Step 8Data quality and observability

    Proving the output is right and noticing when it isn't.

    Practice:Data quality & observability frameworkData engineering · IntermediateData alerting systemData engineering · IntermediateA/B test analytics pipelineData engineering · Advanced

  9. Step 9Reliability, backfills and DataOps

    Pipelines that rerun safely, recover from failure and absorb change.

    Practice:Fraud detection data platformData engineering · AdvancedUser data deletion across a data platformData engineering · Intermediate

  10. Step 10Serving analytics

    Answering dashboards in milliseconds without scanning raw events.

    Practice:Real-time analytics platformData engineering · Advanced

  11. Step 11Practise end to end

    Put every step together under interview conditions.

    Practice:Analytics for a food delivery platformData engineering · Intermediate

What a data engineering design round looks like

You'll be asked something like "design a pipeline that tracks user activity on an e-commerce site", a warehouse for customer history, or real-time analytics for a delivery platform. It isn't a backend round: load balancers and caches matter little, while ingestion, modeling, formats, quality and reruns matter a lot. Strong answers walk six steps (the full framework):

  1. Requirements. Who consumes the data, what they need (aggregates, raw events, snapshots), how they access it, and the latency SLA, volume, availability and retention. Then do the arithmetic: it picks your compute engine and rules out whole categories of tools.
  2. Pipeline design. Batch, streaming, or both (Lambda or Kappa, or a lakehouse), plus the orchestrator.
  3. Data modeling. Medallion layers, star schema vs one big table, and slowly changing dimensions.
  4. Storage and formats. Parquet vs Avro, Delta or Iceberg, and partitioning or clustering.
  5. Quality and observability. Checks per quality dimension, data contracts, and pipeline monitoring.
  6. Scale, backfills and DataOps. Idempotent jobs, safe backfills, and schema evolution.

Interviewers go wide, then zoom into two or three steps. See the food delivery analytics walkthrough for a full answer, and data engineering design patterns for the architectures you'll name along the way.

Core tools by category

CategoryCommon examplesGuide
Storage formatsParquet, ORC, AvroColumnar file formats
Table formatsApache Iceberg, Delta Lake, Apache HudiOpen table formats
Batch processingApache Spark, warehouse SQL enginesHow Spark works
StreamingApache Kafka, Apache Flink, Spark Structured StreamingWindows and watermarks
OrchestrationApache Airflow, Dagster, PrefectPipeline orchestration
TransformationSQL models (for example dbt) running in the warehouseETL vs ELT

Interviewers rarely care which product you pick. They care that you know what the category is for, what it costs, and when you'd choose something simpler.

A six-week plan

WeekStagesPractise
11-2Learn the six-step framework, then draw the end-to-end platform for an e-commerce company and justify each layer, with volume estimates. The online store warehouse round is a free place to start.
23-4Explain a Spark job's stages and shuffles and fix a skewed join; then design the clickstream pipeline with late events and deduplication, and the free change data capture round.
35-6Compare Lambda and Kappa for one problem; model orders and customers as a star schema and as one big table, and load a customer dimension as SCD Type 2. Then try the SCD2 warehouse round.
47-8Choose formats and a layout (partitioning or clustering) for a 75 TB event table; add quality checks per dimension and a data contract. Then the data quality round.
59-10Design the ETL framework with backfills, idempotent writes and schema changes; add a serving layer for a live dashboard.
611Do full mock rounds: the food delivery analytics round and the real-time analytics round, narrating every trade-off.

Frequently asked questions

Do I need to learn Spark and Kafka to become a data engineer?

You should understand the model behind each: distributed batch processing with partitions and shuffles, and a durable, partitioned event log with consumer offsets. Most data teams use one or both, and interviews assume you can reason about them.

Is data modeling still important with lakehouses?

Yes. Cheap storage changed where data lives, not what analysts need. A clear grain, conformed dimensions and history handling are still what make numbers consistent across reports.

Batch or streaming: which should I learn first?

Batch. Most pipelines are batch, the concepts (idempotency, partitioning, backfills) carry over, and streaming adds time semantics on top. See batch vs stream processing.

Frequently asked questions

What should I learn first to become a data engineer?

Start with SQL and data modeling, then batch processing with Spark and an orchestrator such as Airflow. Add streaming (Kafka, Flink or Spark Structured Streaming) once batch concepts like partitioning, idempotency and backfills feel natural.

How long does the data engineering roadmap take?

Working through it with practice takes about six weeks part-time: two stages a week, each followed by a design exercise or practice round.

Is the data engineering system design round different from a backend one?

Yes. Backend rounds focus on APIs, load balancing and caching; data engineering rounds focus on ingestion, batch vs streaming, data modeling, storage formats, data quality and backfills.

Further reading