A successful backup job does not prove recoverability. BitzOrcas business facts, objects, message progress, Data Protection keys, and configuration are related and must be tested on one recovery timeline.

Critical path diagram

Follow the main path to identify responsibility handoffs, then use the prose to inspect failure branches and evidence.

Define the target

RPO: maximum acceptable data loss after failure;
RTO: time from disaster declaration to restored critical service;
scope: one tenant, one database, a region, or a provider;
consistency point: how database, objects, and messages return to an explainable shared point.

Different data can have different targets, but a customer commitment must account for the slowest critical dependency.

Assets to protect

Asset	Minimum requirement
SQL Server/PostgreSQL	Full plus log/incremental backups, regularly restored in isolation
S3/MinIO	Versioning, lifecycle, cross-region/offline copy, and reference reconciliation
Redis	Separate disposable cache from critical Data Protection keys and idempotency state
RabbitMQ/CAP	Persistence, dead-letter/backlog monitoring, and idempotent redelivery after restore
Configuration/secrets	Versioned configuration, secret re-issuance, and access audit
Release artifacts	Preserve images, packages, SBOM, provenance, and migrations as one version

Suggested recovery order

Stop writes and record failure time, last trusted commit, and message positions.
Restore networking, secrets, database, and the Data Protection key ring.
Restore object storage and run reference-integrity checks.
Resume broker, CAP, and background work at a controlled consumption rate.
Start read-only verification for tenancy, authentication, critical queries, and workflow state.
Re-enable writes and watch redelivery, cache rebuild, count drift, and notification backlog.

Restoring the database before objects or messages is not automatically safe. Deletes, workflow completion, and integration events may require replay or compensation, with rules written before an incident.

Rehearsal acceptance

use production-equivalent encrypted backups, not developer snapshots;
restore into a clean environment with no original-machine residue;
measure actual RPO/RTO and record manual steps;
sample multiple tenants, dates, file downloads, login, approvals, and notifications;
convert findings into automation, monitoring, or explicit risk acceptance.

See database migrations and the production security checklist.

SQL Server backup and verification

BitzOrcas.DatabaseMaintenance implements Full, Differential, Log, VERIFYONLY, and confirmed Restore. The backup directory must be visible under the same path to SQL Server and the CLI; a remote server does not download files to the CLI machine.

# ① Create a full backup with a controlled operations identity.
dotnet run --project src/Tooling/BitzOrcas.DatabaseMaintenance -- \
  --backup-database --backup-type Full \
  --connection "$DB_MAINT_CONNECTION" --backup-dir "$BACKUP_DIR"

# ② VERIFYONLY proves readability, not a real restore or application smoke.
dotnet run --project src/Tooling/BitzOrcas.DatabaseMaintenance -- \
  --verify-backup --connection "$DB_MAINT_CONNECTION" \
  --backup-dir "$BACKUP_DIR" --backup-file "$BACKUP_FILE"

Under SIMPLE recovery, a log backup returns code 0 with Skipped=true. Automation must alert on skipped status or it can report a healthy chain that does not exist.

Isolated restore drill

# Run only against an isolated instance; Restore uses WITH REPLACE and has no dry-run.
dotnet run --project src/Tooling/BitzOrcas.DatabaseMaintenance -- \
  --restore-database --connection "$ISOLATED_RESTORE_CONNECTION" \
  --backup-dir "$BACKUP_DIR" --backup-file "$BACKUP_FILE" \
  --database BitzOrcas_RestoreDrill --confirm RESTORE

The tool creates a pre-restore full snapshot. If restore then fails, that snapshot remains. Record partial state and ensure every external connection, broker, email, and webhook in the drill cannot target production.

Reconcile business facts

Scope	Check
Identity and tenancy	multi-tenant login, permission, cross-tenant denial
Files	database references match object versions and download
Messaging	outbox/inbox state, backlog, duplicate consumption
Workflow	instances, todos, timers, timeline
Audit	current and cross-bucket records
License and keys	DeploymentId, cache, Data Protection

Drill report

Record failure hypothesis, backup chain, hashes, SQL version, start/end, actual RPO/RTO, manual steps, failures, samples, and owners. A successful restore that misses customer RTO still fails acceptance.

Completion checklist

backup, VERIFYONLY, and real restore are separate evidence;
database, object, message, key, and configuration times are explainable;
the isolated environment cannot emit production side effects;
tenancy, files, workflow, audit, and messaging have post-restore smoke;
drill findings enter an owned automation backlog with due dates.

Keep the next drill date, owner, and open risks in the same report so one success is never treated as permanent recoverability.