Most of Your Backfills Didn't Have to Happen

A teammate finds a bug that's been quietly corrupting the last month of data. A two-line fix. To clean up the history, they re-run the pipeline over the last thirty days and head to lunch.

They come back to doubled revenue — what a productive lunch huh?? Almost seems too good to be true!

Ok, what actually happened — the job wrote with INSERT in append mode, so re-running it didn't replace the old rows, it stacked new ones on top of them. Month of data, counted twice. Every dashboard downstream now shows a company that overnight started selling twice as much as it did yesterday.

The bug fix was two lines. Cleaning up after the fix, however, was a full day of deleting duplicate partitions, re-running with the right write mode, and explaining to people why the whole weekend's numbers were fiction. The job that was supposed to fix an incident instead became one.

That's a backfill: re-running a pipeline over old data to rewrite partitions that already exist. Every data team does it. Most of us treat it as an unavoidable chore, and I'm not sure if it is.

Schemas change. New events show up. An audit shows up wanting the last fiscal year restated. Sometimes you genuinely have to reprocess history and nobody's at fault. Sure, fine — some backfills are truly unavoidable.

What I don't buy is extending that excuse to all of them. "Necessary evil" is the phrase we reach for so we don't have to ask why this particular backfill exists. If a backfill is unavoidable by definition, then you don't investigate it, and you automatically recompute, reconcile, and close the ticket which is exactly how a bug that wrecked three months of numbers gets written off as maintenance.

So count your last few backfills. Fewer of them were unavoidable than the phrase would have you believe.

Type A, the bug backfill

Something broke, you fix the code, then you re-run history with the fresh fix. The bug itself is fine, we all write them. The real issue is that it lived in production for months and nobody noticed, and when someone finally did, cleaning it up meant a hand-written script. That is the real story, not the bug: not that a bug happened, but that it stayed silent so long and that fixing it hurt.

Type B, the schema-change backfill

You added a column or changed a transform and need the new logic applied to old data. Genuinely unavoidable, because you can't see the future, though if it's happening every month, look closer: you're probably materializing ad-hoc columns upstream that you could be deriving downstream, and each of those dependencies forces another backfill you never needed.

Type C, genuinely new data

You started tracking a new event and want to reconstruct history from logs where you can. Nothing failed here — you just want more history than you used to keep.

B and C are the real necessary evils, and in those cases the goal is to make them cheap. Type A is neither necessary nor evil. It's a symptom.

Sort your team's last five backfills into these buckets. If four are Type A, then it's not about inevitability: almost every one could've been caught in an hour instead of three months, and fixed with a button instead of a script. "Necessary evil" here isn't describing your platform, it's describing how you let yourself off the hook for not fixing the cause.

Take each Type A backfill and ask: what property was missing that made it necessary in the first place? I'll bet the answers all come down to one of the following five.

1. The pipeline wasn't idempotent. You couldn't just re-run the job for the broken window, because a second run would double the data, break dedup, or overwrite state downstream. So you wrote a one-off script instead. A job that gives a different result on its second run is dangerous by itself, and it will get re-run, that's the whole point of a scheduler. One extra run, and there's the story from the top of this post (real story btw).

2. There was nothing to replay from. The data got transformed on the way in, written once, and forgotten. No untouched raw copy you could just re-run the fixed transform against. So instead of replaying, you're reconstructing: piecing state back together from CDC logs, a warehouse snapshot, or a backup you're not fully sure is complete.

3. The schema changed upstream and nothing warned you. A source adds a field. Turns a nullable column into a required one. Your pipeline validated none of it at ingest, so it just kept going: reading and writing, until a human finally noticed the numbers looked off.

4. You couldn't rebuild the downstream tables even with clean input. Because over the years they quietly stopped being a function of their inputs. A manual SQL fix here. A one-off UPDATE there. A "temporary" column somebody added in 2022 that three dashboards now depend on and nobody dares delete. The table became its inputs plus half a decade of tribal edits nobody wrote down, so recomputing from scratch doesn't reproduce it. "It" was never fully in the code.

5. Nobody was actually checking whether the output was right. You knew the job ran. You knew rows got written. What you didn't know was whether any of them were correct. Freshness monitoring is everywhere; correctness monitoring almost nowhere. That gap is why bad data gets to sit in production long enough to need a backfill at all.

Take any partition from three months ago. Ask two things: how long and how many people are needed to regenerate it exactly.

• One command, one person, ten minutes — you've got a real platform.
• One person and an hour with a runbook — healthy, keep going.
• Half a day, two engineers, a Slack thread and a hand-written script — a backfill is an event for you, not an operation, and the next incident is going to be painful.
• "We can't, the source data's gone" or "the logic changed and we didn't keep the old version" — what you have isn't a platform, it's a pile of scripts that produced the right output once.

Some of the teams I've worked with sit squarely in that third case and describe themselves as the second. Run the test honestly. Then keep reading, because the rest of this article is how you move up.