Skip to content

The attacker had administrator access. Why would the backups survive?

The Backup Succeeded. Can We Recover? · Part 4 of 6 · 5 min read

Read the worked explanation ↓

A company with 25 TB of customer records discovers that a privileged account has altered records for four days and erased the newest snapshots. Design protected backup authority, clean-room recovery and reconciliation of legitimate writes.

Think first

A company with 25 TB of customer records discovers that a privileged account has altered records for four days and erased the newest snapshots. Design protected backup authority, clean-room recovery and reconciliation of legitimate writes.

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

Interview question. An attacker gains administrative privileges, deletes production data, and tries to remove all backups. Design a recovery architecture that survives the compromised production administrator while preventing the restored environment from being infected again.

The first mistake in this interview is assuming the attacker is an ordinary buggy application. They have the keys that production operators normally use. If the same identity can delete database clusters, disable backup schedules, remove storage versions, and destroy encryption keys, the backup platform is just another target. Replication faithfully carrying a mass deletion to every region does not save you. Neither does a nightly snapshot that an attacker can erase with one account-level command.

I would begin with the threat boundary. Suppose the attacker can control production administrators, application hosts, and the production cloud account. They do not initially control a separately administered backup account or its offline recovery credentials. That separation is a design assumption, and we must test whether trust relationships actually enforce it. If production roles can assume the backup administrator role through a forgotten cross-account policy, there is no meaningful boundary.

Copy backups into a distinct security domain using narrowly authorized write access. Production can deliver new backup material but cannot rewrite earlier recovery points, shorten their retention, or unilaterally erase them. Use provider-supported immutable retention where appropriate. In AWS Backup, governance-mode vault locking can still be removed by a sufficiently privileged identity, while compliance-mode locking becomes immutable after its configured grace period. That distinction must appear in the interview answer. Protect the backup account's closure authority as well: AWS documents that closing an account can ultimately delete even locked backups after the suspension window. A lock is not a substitute for keeping the backup security domain and its lifecycle outside the production attacker's authority. 'We enabled Vault Lock' is not specific enough to establish survivability.

Production has permission to deposit backups in an isolated retention domain but not to erase locked recovery points while clean-room restore uses separate authority.
Production has permission to deposit backups in an isolated retention domain but not to erase locked recovery points while clean-room restore uses separate authority.

The backup copy must also survive compromised keys and identities. If the only decryption key is controlled and deletable by the attacker, immutable ciphertext can become permanently useless. Define the recovery path for encryption key access, hardware-backed key policy where suitable, emergency credentials, and the people allowed to use them. Restrict who can approve changes to retention and trust policy. Multi-party approval reduces one-person mistakes, but by itself does not replace the cryptographic or provider-enforced denial of destructive operations.

What happens when the attacker changes a hundred million rows slowly, over six days, before finally wiping the database? A clean backup is not necessarily the most recent backup. We need a timeline of compromise: authentication events, unusual privileges, data changes, backup deletions attempted, and confirmed suspicious deployments. Keep relevant audit records away from the compromised account. Identify the last trusted recovery point, not merely the latest successful snapshot. That point may have a worse RPO than the ordinary hardware-failure target because the corruption began much earlier.

A robust restore starts in quarantine. Create a clean account or isolated network environment, establish known-good infrastructure images and deployment artifacts, rotate compromised credentials, and validate restore media. Restore authoritative data and scan for persistence mechanisms, malicious stored procedures, poisoned jobs, unauthorized access grants, or data modifications. Do not reconnect recovered hosts directly to the original production control plane. Otherwise the same actor or credential may immediately delete the newly restored data.

The design also needs a path for legitimate changes after compromise began. A bank cannot simply revert six days of all customer activity. Preserve original logs and payment evidence, identify corrupt versus valid transactions, and build a documented reconciliation pipeline. That is a business recovery problem, not just a storage restore. Some records may be recoverable from unaffected external processors. Others may need customer notification or manual review. Be explicit about cases where no safe automated merge exists.

There is a separate detection problem. An attacker may corrupt invoice amounts rather than delete whole tables, so file-level checksums could remain perfectly valid after the next backup. Keep independent business invariants and historical comparisons, and track when a value first diverged. A clean restore point is a matter of verified content, not just storage integrity.

Retention is not infinite by default. Locking every backup forever is expensive and can conflict with deletion policies. Choose layered retention based on detection horizon: recent frequent recovery points, longer-lived periodic points, and separately protected copies for incidents discovered weeks later. Legal hold and regulated retention create additional constraints. In any case, test that a compromised production administrator cannot erase the protected generations within the promised window.

For operational evidence, run a controlled red-team exercise with a role equivalent to production administrator. Attempt to disable backup schedules, alter vault policy, destroy recovery points, and access key controls. Record what succeeds, what fails, and whether independent alerts arrive. Then restore a selected point in an account with no inherited production trust. A green backup dashboard does not prove any of these boundaries.

The winning interview design is not 'three copies in three regions.' It is a carefully defined adversary, independent authority over protected history, a defensible choice of clean recovery point, and a path to reopen service without carrying the compromise forward. The strongest last question is simple: who can delete the last copy, and how do we prove the attacker is not that person?

Go deeper

NIST SP 1800-11 treats recovery from destructive events as a data-integrity problem, not only a hardware availability problem. Immutability policies have vendor and mode-specific limits. A protected recovery point can still be inaccessible if supporting credentials, KMS policy, catalog metadata, region dependencies or required restore tooling are missing. Independent object copies, tested key custody, minimal production-write permissions and security logging help, but cannot guarantee a clean snapshot when compromise time is unknown. Distinguish logical corruption, malicious identity changes and execution-environment compromise in the runbook.

Check your reasoning

  • Define what attacker-controlled administrator rights can delete or modify.
  • Separate production deposit access from backup retention and deletion authority.
  • Protect encryption keys, audit evidence and recovery identities from the same compromise.
  • Restore in a quarantined environment to a verified trusted recovery point.
  • Reconcile valid activity during compromise and test protections adversarially.

Common mistake: Any backup stored in another region is automatically protected from the production root account.

Which architecture can survive a compromised production administrator and support trusted recovery?

Read the answer explanation

Protect backups and keys with independent authority, immutability and clean-room recovery — Location diversity does not replace independent destructive-operation authority or verified clean recovery.

Apply it elsewhere

A hospital stores encrypted medical records and must preserve legal holds. Plan a ransomware restore while keeping audit evidence intact and preventing a stolen admin account from destroying the remaining copies.

Compare your approach

Specify attacker capabilities, separate deletion authority and encryption access, identify a trusted recovery point, quarantine the restore, reconcile legitimate data and prove resistance with adversarial drills.

One more constraint

Does governance-mode immutability alone prevent an attacker with sufficient management permissions from removing a backup lock?

Quick revision

An administrator-level attacker can erase the same backups that ordinary operators manage. Protect backup history in a separate security domain, with narrowly scoped deposit access, retention-enforced immutability and independently recoverable keys. Select the last trusted point using external audit evidence. Restore into a clean environment, rotate credentials, check for persistence and reconcile legitimate transactions after compromise began. Test the protection using a production-equivalent attacker role, not only a successful backup job.

References