Chapter 18: Recoverability¶
The Restore You Have Actually Run¶
1. What "Backup" Means Now¶
The three structures assume the system can come back. Recoverability is the difference between a copy and a recovery — and the distinction starts with a question most backup plans never ask: what can we rebuild, and what can we not?
| Data | Rebuildable? | So |
|---|---|---|
| generated artefacts, caches, indexes | yes | rebuild them; do not back them up |
| configuration, rules, manifests | yes, if in the repository | the repository is the backup |
| user data, audit records | no | back it up, and prove the restore |
| keys and credentials | no | custody is the recovery |
Backing up what you can rebuild is expensive theatre. Backing up what you cannot rebuild is the whole job.
💰 Cost Check: backing up what you can rebuild costs storage, restore time and a drill nobody finishes. A copy in the same account costs nothing and survives nothing. A key stored beside its data costs the whole backup. The cheap plan is the one that classifies first and stores second.
🎯 What You'll Learn¶
- Which data is irreplaceable and which is derived
- Versioning versus copy-out, and why both are needed
- Key custody: the part that makes a backup decorative
Two Copies, Two Different Failures¶
Versioning survives mistakes: a delete or an overwrite can be undone from history. Copy-out survives account loss: the data exists outside the provider entirely.
A versioning bucket is not a backup. A copy in the same account is not a backup. Both fail the same day, in the same way, and the plan that relied on them looks complete right up until it is tested.
flowchart LR
D["data"] --> V["versioning<br/>survives a mistake"]
D --> C["copy-out<br/>survives account loss"]
V --> R["restore drill"]
C --> R
R --> P["a proven restore<br/>— or a documented gap"]
What this shows: two mechanisms, two failures, and one drill that proves the pair.
✅ What to Check¶
- Which of your data is irreplaceable? Is any of it only versioned?
- Does any copy live only inside the account it protects?
2. Retention as Policy¶
Retention is a policy: daily for a week, weekly for a month, monthly for a year. It is enforced by the store's lifecycle rules — declared, versioned, reviewable — and never by a script that deletes.
A deletion script is a second implementation of a policy the platform already has, and it is the one that runs at 3am and removes the wrong prefix.
The Retention Policy, as a shape¶
# retention — declared, not scripted
daily: { keep: 7 }
weekly: { keep: 4 }
monthly: { keep: 12 }
enforced_by: <lifecycle rule id>
never_delete: <the artefacts that must survive every policy>
✅ What to Check¶
- Is retention a rule in the store, or a script in the repository?
- Does anything you must keep forever have an exception written down?
3. Key Custody¶
If the key that decrypts the backup lives only beside the backup, the backup is decorative. Custody is the recovery — and it is the step most plans omit, because the plan describes copies and not access.
| The question | The record |
|---|---|
| where is the key? | outside the account, in a named store |
| who can use it? | the roles, and the recovery path |
| what if that person is unavailable? | the break-glass procedure, with its own audit |
Break-glass is a control, not a loophole. It is documented, audited, and rehearsed — because the day it is needed is the day the normal path is gone.
✅ What to Check¶
- Does the key exist outside the account it protects?
- Has the break-glass path been used in a drill, and recorded?
The other half of the way back is the release's rollback (chapter 25). A restored copy and a restored deployment are different returns: the deployment costs nothing but a minute, and the copy costs every write since it was taken. Knowing which one an incident needs is the decision chapter 25 makes before the release, not during it.
4. Restore Drills as a Signal¶
RPO and RTO, sized for a small organisation. RPO is how much data you can lose; RTO is how long you can be down. Both are decisions, and both are cheap to state and expensive to discover.
The drill proves two things: the copy is restorable, and the runbook is followable. A restore that works because one person knows the undocumented steps has proven the person, not the plan.
The result is a signal. Last successful restore: 12 days ago is an observable invariant (Chapter 17's table), and it is the one that turns recoverability from a document into evidence.
The Drill Record¶
drill:
at: <timestamp>
from: <which copy — versioning | off-account>
target: <where it was restored>
rto_actual: <how long it took>
broke: [<what the runbook got wrong>]
next: <the fix, and the date of the next drill>
✅ What to Check¶
- When was the last successful restore? Is it a signal, or a memory?
- Did the last drill follow the runbook, or did it need someone who knew the steps?
5. What Not to Back Up¶
The inverse rule matters as much as the rule:
- do not back up what you can rebuild — indexes, caches, generated artefacts;
- do not back up what the repository already holds — rules, manifests, config;
- do not keep a backup whose restore nobody has run — it is a rumour with a bill.
Every artefact you exclude makes the restore faster and the drill cheaper. Every artefact you include without a reason makes the drill longer and the signal weaker.
✅ What to Check¶
- Name three things you back up that you could rebuild. Remove them from the plan.
- Does the restore runbook fit on one page?
6. Build the Workbench¶
Artefacts¶
| Artefact | What it is for | Where it lives |
|---|---|---|
| retention policy | declared, lifecycle-enforced | the store, reviewed |
| copy-out | the off-account copy | a separate account or store |
| key custody record | where the key is, who can use it, break-glass | docs/recovery/keys.md |
| restore runbook | one page, followable | the Playbook |
| drill record + signal | the last restore, published as a reading | the evidence layer |
Decisions¶
| Choice | Default | What changes if you choose otherwise |
|---|---|---|
| what to back up | only the irreplaceable | the rebuildable, which slows every restore |
| copies | versioning and copy-out | one mechanism that fails one way |
| retention | lifecycle rules | a script that deletes at 3am |
| the signal | last successful restore, published | a memory nobody checks |
✅ Checks¶
- A restore from the off-account copy succeeds, on a schedule.
- The runbook is followable by someone who did not write it.
- The last restore's age is a signal, with a threshold.
Sequence¶
- Classify the data — rebuildable or not. Checkpoint: the backup list contains only the second.
- Add copy-out and a lifecycle retention policy. Checkpoint: a copy exists outside the account.
- Record key custody and the break-glass path. Checkpoint: the key is outside, and reachable.
- Run the restore drill and publish the result as a signal. Checkpoint: the age is observable.
🎯 Done When¶
- Every irreplaceable dataset has a copy outside the account.
- The restore has been run by someone following the runbook.
- The age of the last restore is a signal with a threshold.
7. 🔑 Chapter Summary¶
A backup is a copy; recovery is a drill.
- Back up only what you cannot rebuild; the rebuildable is theatre.
- Versioning and copy-out fail differently — a versioned bucket is not a backup.
- Retention is a lifecycle rule, never a deletion script.
- Key custody is the recovery; a key beside its data makes the copy decorative.
- The drill proves the copy and the runbook, and its age is a signal.