Start with the business inputs, not the architecture
The most common DR mistake is architectural: a team picks a DR pattern — usually pilot light, because it sounds appropriately serious — before anyone has written down how much data loss and downtime the business can actually tolerate. The pattern should be the output of that conversation, not the input to it.
Two numbers drive everything:
- RTO (Recovery Time Objective) — how long the business can be down before the outage itself becomes the damage. Measured from the moment of failure to the moment the system is serving production traffic again.
- RPO (Recovery Point Objective) — how much data, measured in time, the business can afford to lose. An RPO of 15 minutes means that after recovery, up to 15 minutes of the most recent transactions may be gone.
These are not IT decisions. They come from whoever owns the cost of downtime and the cost of data loss — finance, operations, legal, depending on the system. A payments ledger might need an RPO near zero and an RTO of minutes. An internal reporting dashboard might tolerate 24 hours of both. Building the same DR pattern for both is either wasteful or negligent, depending on which way you get it wrong.
Get these numbers in writing, from the business owner, before you design anything.
The four strategy tiers
AWS’s standard DR framing groups strategies into four approaches — backup and restore, pilot light, warm standby, and multi-site active/active — ordered by cost and complexity (Disaster recovery options in the cloud, AWS whitepaper). Use those names when you write them down; “hot standby” is a distinct fifth shape in the same document — a full-capacity second Region in active/passive configuration, which AWS notes most customers convert to active/active once they are paying for it anyway. None is universally correct — the right tier is whichever one meets the RTO/RPO at the lowest sustained cost, and the honest answer for most non-critical systems is one of the cheaper tiers.
| Tier | Mechanism | Typical RTO | Typical RPO | Ongoing cost | What it actually requires |
|---|---|---|---|---|---|
| Backup and restore | Periodic backups (snapshots, AWS Backup) to a second region; infrastructure built from scratch on failover | Hours | Hours (since last backup) | Lowest — storage only | A tested, automated restore procedure; IaC to rebuild infrastructure, not just data |
| Pilot light | Core infrastructure (databases, critical config) kept running at minimal size in the DR region; everything else deployed on failover | Tens of minutes to a few hours | Minutes (via ongoing replication) | Low-to-moderate — a small running footprint | Database replication already running continuously; automation to scale up and deploy the rest fast |
| Warm standby | A scaled-down but fully functional copy of the production stack running continuously in the DR region | Minutes | Seconds to minutes | Moderate-to-high — a second environment running at reduced capacity | Load balancer/DNS failover tested regularly; the standby must actually be kept in sync with production changes, including schema and config |
| Multi-site active/active | Both regions serving live production traffic simultaneously | Near zero for most infrastructure disasters — AWS notes there is “no such thing as failover in this scenario” | Near zero via asynchronous replication | Highest — full duplicate capacity, plus cross-region data consistency engineering | Conflict-resolution strategy for writes in both regions; this is a distributed-systems problem, not an infrastructure problem |
The official distinction between the two middle tiers is narrower than most teams assume, and it is worth quoting: “pilot light cannot process requests without additional action taken first, whereas warm standby can handle traffic (at reduced capacity levels) immediately.” Pilot light needs you to turn servers on and possibly deploy non-core infrastructure and then scale; warm standby only needs you to scale.
The jump from pilot light to warm standby to multi-site active/active is mostly a jump in how much you pay to run infrastructure you hope never gets used. The jump from backup-and-restore to pilot light is mostly a jump in operational discipline — continuous replication has to actually be monitored, or it silently stops and you don’t find out until the day you need it.
One constraint that cuts across all four tiers and gets missed in planning: AWS advises using only data plane operations as part of your failover operation, because data planes are designed for higher availability than control planes. Practically, that rules out designs whose recovery path depends on creating resources, scaling an Auto Scaling group, or restoring a backup during the incident — a restore is a control plane operation, which is why AWS recommends restoring periodically on a schedule so an operable data store already exists when you need it. Even a near-zero-RTO active/active architecture inherits a non-zero recovery point for data corruption, because that class of disaster is recovered from backups, not from the other Region.
Be honest with stakeholders about this curve: halving RTO rarely halves cost. Going from warm standby to multi-site active/active for a marginal RTO improvement, on a system that doesn’t need near-zero downtime, is usually money better spent elsewhere — including on testing the tier you already have.
Backup is not disaster recovery
This needs to be stated plainly because it’s routinely conflated: having backups is not the same as having a disaster recovery capability. A backup is a copy of data. Disaster recovery is the demonstrated ability to restore service — infrastructure, configuration, networking, DNS, and data — within the RTO, under conditions that resemble an actual incident (compromised credentials, a down region, a corrupted dataset you don’t notice for days).
An untested restore is a hypothesis, not a control. Plenty of organizations discover, during a real outage, that their backup job had been silently failing for months, that the restore procedure assumed a person who left the company, or that the restored database was application-consistent but the message queue it depended on was not — producing a restored system that starts up and immediately corrupts data because it’s missing in-flight state.
AWS takes the same line — “it is critical to regularly assess and test your disaster recovery strategy so that you have confidence in invoking it” — and a backup strategy that does not include testing its backups is incomplete by AWS’s own framing. The only way to know a DR plan works is to exercise it on a schedule, under conditions that approximate the real thing:
- Tabletop exercises quarterly — walk the runbook without executing it, to catch missing steps and stale assumptions.
- Partial restore tests monthly or quarterly — actually restore a backup to a scratch environment and validate the data, not just that the job “completed successfully.”
- Full failover tests at least annually for anything above backup-and-restore tier — actually fail over, serve real or shadow traffic from the DR region, and fail back. This is the test that finds the DNS TTL nobody updated and the IAM role that only exists in the primary region.
Each test that fails is cheaper than the incident it would have failed during. Treat test failures as the system working as intended — finding the gap before it costs you an outage.
Snapshot vs. AWS Backup vs. application-consistent backup
These are not interchangeable, and picking the wrong one for a given workload is a common silent failure:
- Volume-level snapshots (EBS, RDS automated snapshots) capture the disk state at a point in time. For a database, this is crash-consistent, not necessarily application-consistent — it’s equivalent to pulling the power cord. Most modern databases recover cleanly from crash-consistent snapshots via write-ahead log replay, but verify this for your specific database engine and don’t assume it for every data store (in-memory caches, some NoSQL configurations, and multi-file application state are common exceptions).
- AWS Backup is a centralized policy layer on top of the same underlying snapshot mechanisms — it adds consistent scheduling, cross-region/cross-account copy, retention lifecycle, and a single pane for backup compliance across supported services, which include EBS volumes, EC2 instances, RDS and Aurora, DynamoDB, EFS, Storage Gateway volumes and the FSx file systems (Disaster recovery options in the cloud, AWS whitepaper); check current coverage before assuming a given service is in scope. It doesn’t change the consistency guarantee of the underlying snapshot; it standardizes how that snapshot is scheduled, copied, and retained. On restore: AWS Backup does provide scheduled restore testing, which periodically runs restore jobs so you find out whether your backups are actually recoverable before an incident does it for you (Restore testing, AWS Backup docs). Note what it is for — it validates recoverability and then deletes the restored resource. If what you want is a standing, persistent restored copy sitting in the recovery Region, that is still something you script yourself.
- Application-consistent backup means the application was quiesced, or flushed its buffers and committed transactions, immediately before the snapshot — required for databases that don’t tolerate crash-consistent recovery cleanly, and for multi-component systems where the data is spread across more than one storage volume (a common case: a database and a search index that must agree with each other).
The decision criterion: if a single unexplained crash recovery would be acceptable in production, crash-consistent snapshots are fine. If it wouldn’t be, you need an application-consistent mechanism, and AWS Backup’s scheduling layer doesn’t give you that by itself — it has to be built into the application or orchestrated separately.
Cross-region and cross-account copies: the ransomware case
Backups stored in the same account and region as the production data they protect are a single blast radius. A compromised root credential, a misconfigured IAM policy, or ransomware that has obtained write access to the account can delete or encrypt both the production data and its backups in the same action.
The mitigation is cross-account, cross-region backup copies, ideally to an account with no standing write access from production workloads — a dedicated backup/archive account with its own tightly scoped IAM boundary. AWS Backup supports cross-account and cross-region copy as a policy, and S3 Object Lock provides immutability. The mode matters: in compliance mode “a protected object version can’t be overwritten or deleted by any user, including the root user in your AWS account,” the retention mode cannot be changed, and the retention period cannot be shortened — AWS states the only way to delete such an object before its retain-until date is to close the AWS account. In governance mode, a principal holding s3:BypassGovernanceRetention can override the lock, which makes it a control against accident rather than against a sufficiently privileged attacker (S3 Object Lock, AWS docs). Object Lock requires versioning on the bucket, and it protects object versions — it does not prevent new versions or delete markers being written on top. For ransomware and account-compromise scenarios specifically, immutability is the control that matters; encryption and access control alone don’t stop an attacker who has already obtained the permissions to delete.
Retention policy should reflect realistic detection windows — some compromises aren’t discovered for weeks. A 7-day backup retention window is of limited use against an attacker who sat quietly in the environment for a month before triggering damage.
The recovery runbook
Write the runbook before the incident, because the incident is the worst possible time to write it. A usable runbook names specific people and roles (not “the on-call engineer,” but who that is and how they’re reached), specific AWS console and CLI steps in order, the DNS and connection-string changes required, and the validation steps that confirm recovery actually succeeded — not just that infrastructure came up, but that the application is correct.
Keep it outside the systems it’s recovering — a runbook stored only in the production wiki that goes down with the primary region is not a runbook.