GA cutover runbook

This is the operator-facing procedure for promoting a signed BitzOrcas candidate to GA in a target environment. It assumes the production-readiness manifests are current and that optional adapters stay disabled unless their production prerequisites are explicitly satisfied. A ProductionBlocker in 0002-production-adapter-readiness.json stops the cutover.

1. Pre-cutover gate

Before scheduling the window, confirm: release build (0 errors), XML files/comments checks pass, architecture + application + API-shell + Docker-contract tests pass, report-acceptance.sh hard gates pass, git diff --check clean, CI fast-gate and integration-docker configured, verify-template.sh passes, and the signed candidate artifact + provenance.json pass --artifact-only verification against the trusted release/attestation workflow at the exact candidate commit.

The release workflows share one non-cancelling concurrency group. Before the first commercial promotion: protect the commercial-release-builder environment, configure the signing-cert secrets and the trusted-signer/vulnerability/license variables, and record feed staging/promote evidence or an exact-hash resumability drill.

Hard stop conditions: an absent value is a release stop — repository code alone is not evidence that a real cert, timestamp authority, private feed, or successful protected workflow run exists. Do not cut over with a ProductionBlocker, and do not announce Chat realtime unless Chat__Realtime__Enabled=true is deployed and Operations reports SignalRChatRealtimeAdapter.

2. Configuration and secrets

Load production values from the deployment secret store. The required fail-closed baseline:

ASPNETCORE_ENVIRONMENT=Production
RateLimiting__Enabled=true
Webhook__Delivery__Enabled=false
Webhook__Delivery__IpAllowlist__Enabled=false
Webhook__Delivery__RateLimit__Enabled=false

Required production dependencies: SQL Server (ConnectionStrings__Default or SqlSugar__ConnectionString), RabbitMQ, Redis, S3-compatible file storage (FileStorage__DefaultProvider=Minio), the OTLP collector endpoint (OTEL_EXPORTER_OTLP_ENDPOINT), and JWT signing material from the secret store only. Never commit resolved secrets, production .env files, certificates, or private keys.

3. Migration and data preparation

Confirm the DB target is production and backups are current. Never use EF Core EnsureCreated, automatic --reset-schema, or BITZORCAS_ASPIRE_RESET_SCHEMA in Staging or Production. Run required seed steps only after schema initialization or a versioned migration succeeds. Verify CAP tables exist and RabbitMQ is healthy before starting the Web runtime. Record the deployed commit SHA, template version, DB backup identifier, migration current version/checksums, and config version.

EF Core release sequence

The migration process needs only Persistence__Provider=EfCore and ConnectionStrings__Default — do not inject RabbitMQ, JWT, or Runtime License solely to make the DB command start.

Attribution: --migrate-schema plan|status|apply belongs to the template-generated customer host (BitzConsumer.Api). The mainline BitzOrcas.Api only supports —init-schema/—seed-demo/—seed-only/—reset-schema/—init-quartz-schema. Run this section against the generated consumer host.

# Treat plan, approval, apply, and the final status check as separate evidence points.
dotnet BitzOrcas.Api.dll --migrate-schema plan     # review, pending is allowed, exit 0
dotnet BitzOrcas.Api.dll --migrate-schema status    # gate: 0=current, 4=pending, 5=drift
dotnet BitzOrcas.Api.dll --migrate-schema apply     # only after plan approval + backup
dotnet BitzOrcas.Api.dll --migrate-schema status    # must return 0 before Web starts

Logs carry migration id, name, state, and SHA-256 only — never SQL text or the connection string. On exit 5, do not edit an applied SQL resource or rewrite dbo.__BitzOrcasEfSchemaHistory; investigate the DB error, restore the reviewed script, and rerun status before deciding whether another apply is safe.

SqlSugar release sequence

Use the --init-schema --no-seed flow with DBA-reviewed scripts/database/migrations/. SqlSugar CodeFirst is idempotent table initialization, not an EF migration history; Staging and Production still reject automatic reset. --init-schema only creates missing tables and nullable no-default columns. Remaining shape drift is reviewed at /host/schema; the API Host operations-schema-drift-notify loop pushes residuals to Host-Admin. If schema init fails, stop the cutover and keep traffic on the previous version.

4. Health checks

After deploying the new version, before routing user traffic:

Readiness must expose dependency status for database, RabbitMQ/CAP, Redis, file storage, and Webhook production delivery (when enabled). A licensed Host also exposes GET /health/license; see Runtime License.

5. Operations verification

Using an authenticated Operations caller:

Any critical InMemory*, Memory*, Null*, or Unavailable* adapter in production is a release stop unless an explicitly documented fail-closed constraint applies.

6. Webhook operations drill

Run before setting Webhook__Delivery__Enabled=true. The drill covers: missing config (readiness unhealthy, names missing keys), CIDR rejection (delivery denied fail-closed with a structured warning), Redis unavailable (delivery denied fail-closed), limit triggered (first allowed, second denied), and healthy enablement (readiness healthy). Each enabled subscription must carry explicit valid CIDR/IP allowlist entries; empty or invalid allowlists deny delivery.

Production enablement requires:

Webhook__Delivery__Enabled=true
Webhook__Delivery__IpAllowlist__Enabled=true
Webhook__Delivery__RateLimit__Enabled=true
Webhook__Delivery__RateLimit__PermitLimit=<positive-integer>
Webhook__Delivery__RateLimit__WindowSeconds=<positive-integer>

7. Traffic cutover

1. Deploy the release artifact with production config and secrets mounted.
2. Wait for /health/live and /health/ready to pass.
3. Verify Operations adapter state and config diagnostics.
4. Route a small percentage of traffic to the new version.
5. Watch HTTP 5xx rate, /health/ready, CAP outbox backlog, Redis connectivity, Webhook denied logs, and audit write health.
6. Increase traffic in controlled steps only while gates stay green.
7. After full cutover, keep the previous version and DB backup available until the rollback window closes.

8. Rollback

Triggers: /health/ready fails for three consecutive probes; HTTP 5xx rate exceeds the severe threshold (>5% per 5min); CAP outbox backlog crosses the severe threshold (>1000 rows); Redis failure disables the HMAC nonce store or Webhook limiter; the adapter guard reports unexpected default adapters; or a migration produces correctness/tenant-isolation risk.

Steps: stop increasing traffic and route all traffic to the previous healthy version; keep the failed version running only if needed for diagnosis and not writing unsafe data; if the incident is delivery-specific, set Webhook__Delivery__Enabled=false; preserve audit logs, the acceptance report, deployment logs, and health snapshots; restore data only from an approved backup when the incident owner confirms corruption or incompatible schema writes; record the incident with commit SHA, config version, first failing health check, and rollback time.

9. Incident handling

10. Post-cutover record

Record: commit SHA + artifact identifier; acceptance report path; CI run URL; DB backup identifier; config version + secret-store version; health snapshots before and after; the Operations adapter report; and Webhook delivery status (including whether production delivery remains disabled or has passed the drill).

See also

• Production readiness
• Alert runbook
• Commercial GA gate
• Database migrations