Alert runbook

The BitzOrcas template defines alert thresholds but does not auto-trigger alerts. An external monitoring system — Prometheus, Datadog, Azure Monitor, or Grafana — scrapes OpenTelemetry metrics and fires the alerts against these thresholds. This page is the reference threshold table and per-signal runbook; Monitoring and alerting covers the generic observability setup.

Alert thresholds

Per-signal diagnostic runbook

The common shape: read the Operations adapter/config endpoint, inspect the named subsystem, check dependent infrastructure (SQL/RabbitMQ/Redis), remediate config, restart or rolling-redeploy, and confirm recovery via the health probe.

/health/ready consecutive failure

1. GET /api/operations/adapters — identify which readiness-tagged health check failed.
2. GET /api/operations/config — confirm the connection string and RabbitMQ config.
3. Connect to SQL Server and run SELECT 1 to confirm reachability.
4. Check the RabbitMQ management UI for connection count and queue state.
5. If the dependency recovers, Kubernetes auto-resumes traffic distribution.

HTTP 5xx error rate spike

1. GET /api/audit?Category=Exception&From=<5min-ago> — read the restricted envelope, then correlate logs and traces by CorrelationId for details.
2. Inspect the OTel trace to find the top error endpoint.
3. For an EF Core/SqlSugar exception, check the DB connection pool and deadlocks.
4. For OOM, check dotnet-counters GC and heap size.

CAP outbox backlog

1. RabbitMQ management UI — confirm the bitzorcas.* exchange has consumers.
2. SELECT COUNT(*) FROM Cap.Published WHERE Retries < 50 — view the backlog volume.
3. If consumers are not registered, check that the API/JobHost started correctly.
4. If RabbitMQ is unreachable, CAP auto-retries; the backlog drains on recovery.

Job consecutive failure

1. GET /api/audit?Category=BackgroundJob&Module=Auditing — read job outcomes and stable error codes, then correlate logs for exception details.
2. If IAuditRetentionPort is not registered, confirm AddBitzOrcasSqlSugarAuditStore was called.
3. For an SQL execution timeout, check the audit table index and data volume.
4. Manual rerun via scripts/database/seed-demo.sh (seed problem) or a JobHost restart.

InMemory/Null adapter in production

1. GET /api/operations/adapters — confirm which ports are Default/Unavailable.
2. GET /api/operations/config — confirm ConnectionStrings:Default and RabbitMq:Host are configured.
3. If config is missing, inject via environment variables and rolling-restart.
4. If config is correct but the adapter is still Default, inspect the branch logic in PersistenceRegistration.

Missing/weak key config

1. GET /api/operations/config — find the missing or too-short keys.
2. Inject the correct value via the K8s Secret / Key Vault.
3. Rolling-restart.

Evidence capture before and after remediation

Preserve base health, adapter, and configuration diagnostics first. Capture all three from the same instance and time window, and keep secret fields out of the incident attachment:

# Base readiness and runtime-composition snapshot.
curl -fsS https://<HOST>/health/ready > ready.json
curl -fsS -H "Authorization: Bearer <TOKEN>" \
  https://<HOST>/api/operations/adapters > adapters.json
curl -fsS -H "Authorization: Bearer <TOKEN>" \
  https://<HOST>/api/operations/config > config-diagnostics.json

After recovery, repeat the audit capture over the same window. The query endpoint has no Result parameter; aggregate returned Success and ErrorCode values instead of assuming an unknown query key was applied:

# Keep the incident boundary so before/after job evidence remains comparable.
curl -fsS --get -H "Authorization: Bearer <TOKEN>" \
  --data-urlencode "category=BackgroundJob" --data-urlencode "module=Auditing" \
  --data-urlencode "from=<INCIDENT-START-UTC>" --data-urlencode "pageSize=200" \
  https://<HOST>/api/audit > background-jobs.json

Monitoring-system integration

See also

• Monitoring and alerting
• Observability
• GA cutover runbook