Data Pipeline Orchestration: Airflow, DAGs and Backfills

Data pipeline orchestration is the scheduling and coordination of the jobs that move and transform data: running them in the right order, on time, with retries, and with a record of what ran. Pipelines are modelled as a DAG (directed acyclic graph) of tasks, and an orchestrator such as Apache Airflow, Dagster or Prefect decides when each task can run. The design questions are less about the tool and more about making tasks idempotent, dependencies explicit, and backfills safe.

DAGs, tasks and runs

• A task is one unit of work: run a SQL model, launch a Spark job, call an API.
• A DAG run executes the graph for one logical date or data interval, for example "the data for 2026-10-16". That interval is not the same as the time the run actually starts.
• The scheduler starts a task when its upstream tasks succeed, and retries failed tasks with backoff.

Make tasks idempotent

Orchestrators retry, operators re-run, and backfills rerun history. A task must produce the same result whether it runs once or five times for the same interval.

• Write by partition and overwrite it: a run for 2026-10-16 replaces that day's partition instead of appending.
• Or MERGE on a key when the target is updated in place.
• Parameterise by the run's interval, never by "now". A task that reads current_date produces different results when rerun tomorrow.
• Publish atomically: write to a staging location and swap, or commit through a table format, so readers never see half a partition.

Blind appends are the most common orchestration bug: a retry after a partial failure double counts a day's revenue, and nobody notices until finance reconciles.

Backfills

A backfill re-runs a pipeline for past intervals, for example after fixing a bug or adding a column. With idempotent, interval-parameterised tasks, a backfill is just many DAG runs for older dates. Plan for:

• Concurrency limits so a year of backfill doesn't overwhelm the warehouse or source systems.
• Ordering when each day depends on the previous one (running balances, SCD loads).
• Source availability: you can only backfill from data you still have, which is why raw history is kept in a bronze layer.
• Downstream effects: consumers of the corrected tables may need to rerun too; lineage tells you which.

Scheduling: time-based, data-aware and event-driven

Airflow models pipelines as task DAGs and has added dataset-based scheduling. Dagster centres on software-defined assets: you declare the tables you want and how each is computed, and it derives the graph. Prefect emphasises Python-native flows with dynamic tasks. All of them can run the same well-designed pipeline.

Operating pipelines

• Alert on data SLAs, not just failures: "fact_orders for yesterday is not ready by 06:00" catches stuck sensors and silent delays.
• Keep orchestration light: the orchestrator schedules work; heavy compute runs in Spark, the warehouse or containers.
• Separate environments: test DAG changes on staging data before production.
• Version DAG code and deploy it like application code, with review and CI.
• Gate publication on [quality checks](/learn/data-quality), so bad data stops before consumers see it.