Recovery-First: Design the Restore Before the Deploy

Most infrastructure is designed forward: stand up the cluster, deploy the apps, and then — maybe — bolt on backups. I’ve come to think that’s backwards. The most honest test of an architecture is whether you can rebuild it from nothing but a Git history and an off-host backup. If you design the restore path first, everything else gets healthier almost by accident.

What “recovery-first” forces you to do

When the restore is the requirement, a few rules stop being optional:

Nothing in the cluster can be the root of trust. If your secrets, your registry, or your Git server only exist inside the thing you’re trying to recover, you don’t have a recovery plan — you have a circular dependency. The recovery roots have to live off-host.
State has to be explicit and exportable. You learn very quickly which data is precious and which is disposable, because you have to name the precious bits and ship them somewhere safe, encrypted.
The rebuild has to be written down and rehearsed. A runbook you’ve never executed is a hypothesis, not a plan. The first restore drill is always humbling; that’s the point of doing it before you need it.

Layers, not a single backup

A single backup is a single point of failure. I prefer independent layers that fail differently: a retain policy so a careless delete doesn’t destroy data; a resource-level backup of the cluster’s declared objects; and an application-level, encrypted, off-host backup whose restore path doesn’t depend on the cluster existing at all. Any one of them can be broken and you’re still fine.

The payoff

Recovery-first design has a pleasant side effect: it makes the normal days calmer too. Immutable artifacts, declarative state, and off-host roots aren’t just good for disasters — they’re what make routine changes safe to reason about. You stop being afraid of the infrastructure, because you know you can always get back to a known-good state.

If you can’t confidently answer “what happens if this machine dies tonight?”, that’s the most valuable piece of work you could pick up next.