In every technical assessment I do there is one question with an immediate answer: are there backups? Yes, of course, daily.
The second question produces the pause: when was one last restored? And how long did it take?
That second question is the only one that counts. A backup nobody has restored is not a backup, it is an assumption. This article describes how a rehearsal works, what it regularly turns up, and what rhythm is realistic.
Why "are there backups" says nothing
A backup is a chain of several steps, and it breaks silently at every one of them. The job creating it reports success, and nobody checks what is in the file.
Three kinds of silent failure I have seen more than once, and none of them produced an alert.
The backup is empty or incomplete. An export whose credentials had stopped working wrote a file containing an error message for months. The file was there, it had a size, monitoring checked for existence.
The backup does not contain everything. An export with a list of tables written years ago. New tables appeared, the list stayed. The discovery happens at restore time, not before.
The backup is not readable. Encrypted with a key that sat on the server that failed. Or in a format needing a version that no longer exists.
All three are failures in the chain, and all three surface within an hour if somebody attempts a restore. Without an attempt they surface on the day you need them.
What a rehearsal actually measures
The rehearsal answers four questions, and three of them cannot be read out of any configuration.
Can it be restored at all? The yes-or-no question. It is the most important one and the one that fails most often.
How long does it take? That is the actual recovery time, and it is almost always longer than the assumed one. With larger data volumes, loading the data is the smaller part; the time goes on fetching, unpacking and the work afterwards.
How old is the data afterwards? The actual data loss. A nightly backup at two in the morning means, for an outage at five in the afternoon, fifteen hours of loss. That is a number a board can evaluate, and it rarely gets stated that plainly.
Does the application work afterwards? The question most often forgotten. A restored database is not a running system: files are missing, configuration, access, search indexes.
The last two questions are why a rehearsal does not stop at the database.
How the rehearsal runs
The sequence is unspectacular and takes half a day at medium size. What matters is writing it down while it runs, because that record is the value afterwards.
- Note the time. Do not skip this. Without a start time there is no number at the end, only a feeling.
- Fetch the backup, the way you would in an emergency. Anybody who has to retrieve it from archival storage measures hours rather than minutes here, and those are exactly the hours missing from every plan.
- Restore into a new environment, not over the existing one. A rehearsal that overwrites production is not a rehearsal, it is an incident.
- Start the application against it. With the restored database, the restored files and a configuration pointing at the new environment.
- Exercise five business paths. Log in, find a record, create one, produce a report, call an interface. No more, but those five properly.
- Note the time and write down what was missing. That is the result.
The third point is where rehearsals get skipped, because nobody has a second environment. In a cloud environment that is a matter of minutes and a few euros; in your own data centre it is the reason the rehearsal has been pending for years. Then the smaller version beats none: restore the database on a development machine and at least answer questions one to three.
What it regularly turns up
The findings are remarkably uniform. Four things appear in almost every first rehearsal.
Something is missing that is not in the database. Uploaded files, a search index, caches holding content that exists nowhere else. The database gets backed up because it is obviously important; the directory with customer documents gets forgotten, because it does not look like data to anybody.
The application does not start without manual work. A configuration file that lived on the old server and not in the repository. A key needed to decrypt stored data. A certificate. Those things are individually small and collectively the difference between two hours and two days.
It takes longer than expected, in an unexpected place. Not loading the data, but rebuilding the indexes afterwards. Or downloading from archival storage on a low tariff with hours of lead time.
Nobody knows who is allowed to. One person has the credentials for the backup storage, and they are on holiday. That is not a technical finding and in an emergency it is the most expensive one.
Every one of those is a detail when found on a quiet Tuesday. Every one is a catastrophe when found at three in the morning.
A rehearsed restore incidentally provides the data set a test slice can be shaped from: Test data from production.
A rhythm that can be sustained
An annual exercise is better than none and too rare to notice changes. A monthly full rehearsal is thorough and gets skipped after the third time.
What holds in practice is a staggering by effort.
Daily, automatic: the backup gets checked for content. Not for existence. A script looks inside the file, counts the tables or the rows of a lead table and reports when the number deviates. That catches the first two silent failures above.
# Does the file contain anything? Check content rather than size.
tables=$(zcat backup-$(date +%F).sql.gz | grep -c "^CREATE TABLE")
rows=$(zcat backup-$(date +%F).sql.gz | grep -c "^INSERT INTO \`orders\`")
if [ "$tables" -lt 40 ] || [ "$rows" -lt 1 ]; then
alert "backup looks wrong: $tables tables, $rows order rows"
fiQuarterly, by hand: the full rehearsal. With the application, with the five paths, with timing. Half a day, a date in the calendar, one person by name.
After every significant change: the rehearsal again. A new table, a new directory, a change of database version, a migration. That is the point at which a working backup quietly stops working.
The number from the last rehearsal belongs somewhere visible, with a date. Not in a document, but where the operational figures live. "Restore last verified 14 Oct 2026, took 3 h 40 min" is a statement somebody can plan with.
A way back in a release is the same promise, on the operations side: Blue/green and canary without Kubernetes.
Three special cases
Encrypted backups. The key must not live only where the system that might fail lives. That sounds obvious and is the most common design flaw with encrypted backups. So the rehearsal includes fetching the key from its intended source, not from the tester's password manager.
Object storage with versioning. Where files sit in object storage with versioning and lifecycle rules, backup is technically solved. The question then shifts to deletion: can somebody with access delete everything, versions included? If so, the backup is protected against failure and not against an attack.
Managed databases. Automatic backups and point-in-time recovery are there, and that is a real gain. What remains is the timing question: a restore creates a new instance, and how long that takes for your data volume is something you only know once you have done it. The rehearsal gets shorter, not unnecessary.
And what applies in all three cases: the rehearsal tests the route somebody would take in an emergency, not the most convenient one. Anybody using access for the exercise that would not exist at night has tested something else.
If that rehearsal is meant to become a practised procedure, with recovery times you can commit to: how I build resilience into existing environments is on its own page. What is needed beyond that when an entire region fails is a different order of magnitude and covered in its own article.
This article belongs to a series about systems that already exist. The retrospective orders every article in it by situation.
