Data & Retrieval

Backfill

Running a pipeline over past data to populate or correct history — after adding a metric or fixing a bug.

A backfill is running a data pipeline over historical data to populate or recompute the past — when you add a new metric, fix a transformation bug, or onboard a new source and need the history, not just data going forward.

Worked example: after adding a “profit margin” column to your daily aggregates, you backfill the last two years by re-running the pipeline for each past day so historical dashboards are complete. Gotcha: backfills are heavy and risky — they can overload source systems and double-count if the pipeline is not idempotent — so they are usually run in bounded batches (day by day), rate-limited, and designed so re-running a period overwrites rather than appends; idempotency is what makes a backfill safe to retry.