Operations Security, Testing, Runbooks, and Commercial GA

Operations collects some of the platform’s most dangerous capabilities. GA cannot be judged by class or endpoint count. It must prove runtime routing, exact permission, fail-closed approval, explainable execution, rehearsed database behavior, and durable evidence.

1. Risk ladder

Higher risk requires stronger authorization, approval, SoD, plan binding, mutual exclusion, audit, and recovery evidence. The current implementation does not increase every control consistently.

2. Nineteen permissions

Operations owns the website analytics permissions although Website maps those routes. Audit both owned permissions and consuming endpoints.

3. Two-layer authorization drift

Handwritten routes use RequirePermission(OperationsPermissions.*). Requests also pass ResourceDescriptor + AuthorizationAction through Mediator authorization.

Every route change needs a test that both layers derive the same permission. Restore, for example, is backups + Delete + restore sensitivity. Divergence can produce contradictory decisions or wrong audit classification.

4. Approval degradation

Production/Staging requires a ticket. With an OpsExtension Store, the Incident must be Resolved or Closed.

A required environment without the Store accepts any non-empty ticket. Other environments return true immediately.

Tickets are not bound to operation type, database, script hash, backup hash, tenant, reason, or expiry. One closed incident can currently be reused across sensitive actions.

5. Audit matrix

Archive and cache-invalidation audit are now fail-closed, but backup/restore and job audits remain best-effort. The claim that every dangerous action has immutable audit evidence still does not hold.

6. Runtime route gate

The handwritten group maps 9 routes (4 job management, 3 archive, 2 cache governance). Application contains 16 nested [GenerateEndpoint] declarations, but the generator visits namespace-level types only.

GA must enumerate a running EndpointDataSource or OpenAPI document and verify method, path, authentication, exact permission, rate limit, timeout, and response contract. Attribute grep is secondary evidence.

7. Current test evidence

Current tests cover:

• API Shell permissions and responses for governance, adapters, jobs, and config;
• job edits, start/stop, execution, and audit;
• job catalog/executor closure;
• current-tenant archive boundaries;
• Operations runtime-surface manifest;
• configuration declaration alignment with production guards;
• adapter-matrix consistency;
• exact Website analytics dead-letter permissions;
• sensitive adapter/config responses remaining useful after masking.

8. What current tests do not prove

• runtime schema/backup HTTP reachability;
• real SQL Server drift and FullForce execution;
• preview/apply TOCTOU behavior;
• dual-instance migration or same-second backup races;
• verify-before-restore, chain validation, or isolated restore;
• fail-closed approval without OpsExtension;
• mandatory SoD;
• scheduler convergence after edits;
• cold storage and delayed deletion;
• compliance compensation after audit-sink failure.

9. High-value route contract

// ① Inspect a real TestServer EndpointDataSource, not source attributes.
var routes = endpointDataSource.Endpoints
    .OfType<RouteEndpoint>()
    .Select(x => (Pattern: x.RoutePattern.RawText, Methods: x.Metadata
        .GetMetadata<HttpMethodMetadata>()?.HttpMethods))
    .ToList();

// ② Every commercial route must appear exactly once.
routes.Count(x => x.Pattern == "/api/operations/backups/restore")
    .ShouldBe(1);

// ③ Send anonymous, wrong-permission, and correct-permission requests.

This detects both missing generation and ambiguous double mapping.

10. Approval-intent target model

var intent = new SensitiveOperationIntent(
    Operation: "Schema.FullForce",
    Target: databaseFingerprint,
    PayloadHash: migrationScriptHash,
    RequestedBy: actorId,
    ExpiresAt: clock.UtcNow.AddMinutes(30));

// ① Validate status, separate approver, intent, and expiry.
var approval = await approvals.ValidateAsync(ticket, intent, cancellationToken);
if (!approval.IsApproved) return Result.Failure(approval.Error);

// ② Recompute before execution and require new approval on mismatch.

This is a GA target, not current source behavior.

11. Operational metrics

At minimum expose:

• request/result/latency per capability and error code;
• approval required/denied/degraded;
• schema drift, preview hash, executed/skipped/failed statements;
• backup duration/bytes/skipped/invalid verify/restore result;
• job desired-versus-observed, manual execution, duration, failure;
• archive rows/batches/lag/cold-storage pending;
• audit sink failures;
• adapter/config report freshness.

Labels must not include SQL, absolute paths, ticket body, or tenant secrets.

12. Troubleshooting order

Route 404: inspect runtime endpoints, then the nested-type generator limitation.

403: distinguish HTTP permission, Mediator resource/action, and approval gate; do not blindly grant more admin rights.

Schema HTTP 200 but incomplete state: inspect SkippedStatements for per-command errors.

Successful log backup with no file: inspect Skipped and SQL Server recovery model.

Restore without SoD: inspect environment, OpsExtension Store, and EffectiveUserId.

PUT succeeds but GET is unchanged: the current concrete GET projects the code catalog.

13. Release gate

• runtime route inventory maps one-to-one with permission catalog;
• sensitive operations fail closed when OpsExtension is unavailable in required environments;
• approval is non-reusable and intent-bound;
• audit failure has alerting, buffering, or blocking policy;
• schema preview/apply/redetect and partial-failure recovery run on a cloned database;
• isolated restore drill measures RPO/RTO;
• job desired/observed convergence carries version evidence;
• archive, cold storage, delayed delete, and query path are rehearsed;
• logs and responses expose no path, connection, SQL secret, or ticket-sensitive content.

14. P0 GA backlog

1. Fix nested GenerateEndpoint discovery and gate route uniqueness.
2. Add immutable schema plan/hash/fingerprint/lease.
3. Add structured statement results and failure policy.
4. Bind approval to intent/expiry/SoD and fail closed without backend.
5. Require restore verification and bind its evidence.
6. Prevent concurrent backup filename overwrite.
7. Model job desired and observed state.
8. Complete success/failure/rejection audit for sensitive actions.

15. P1 GA backlog

1. Database-provider capability matrix.
2. Encrypted off-site backup, legal hold, and chain-aware retention.
3. Isolated restore and automated recovery drills.
4. Scheduler version, ETag, reload acknowledgement, and kill switch.
5. Cold-storage object adapter, checksum, query, and deletion proof.
6. Merge active adapter health with profile readiness.
7. Configuration schema and cross-key semantic validation.
8. Signed operational evidence export and retention.

The detailed work is mirrored in Architecture-Hub’s Operations Commercial GA Backlog.

16. Current release judgment

Governance, adapter/config/tenant queries, manual job management, and tenant archive can serve controlled internal operations. Schema and backup have Application and Infrastructure code but should not be marketed as a complete database control plane until routing, approval, plan binding, concurrency, and recovery drills are complete.

17. Verification commands

# Application behavior for queries, jobs, and archive boundaries.
dotnet test tests/BitzOrcas.Application.Tests \
  --filter 'FullyQualifiedName~Operations|FullyQualifiedName~BackgroundJobManagement|FullyQualifiedName~ArchiveTenantBoundary'

# Governance, adapter, and job intake constraints.
dotnet test tests/BitzOrcas.Architecture.Tests \
  --filter 'FullyQualifiedName~Operations|FullyQualifiedName~AdapterMatrix|FullyQualifiedName~BackgroundJobIntake'

# Runtime API Shell and handwritten job-management routes.
dotnet test tests/BitzOrcas.Integration.Tests \
  --filter 'FullyQualifiedName~ApiShellTests|FullyQualifiedName~BackgroundJobManagementApiTests'

Operations overview · Background jobs