Monitoring, alerting, and runbooks

Monitoring answers “what is the system doing,” alerting answers “does a human need to act,” and a runbook answers “what should they do first.” Derive all three from the same service objective and failure model instead of alerting on whatever metric happens to be easy to collect.

Signal to action

Minimum SLI set

Thresholds come from measured baseline, capacity, and SLO. This guide deliberately does not hard-code one environment’s percentile or queue count. Keep production values in monitoring configuration and recalibrate after capacity or version changes.

Kubernetes probe example

# ① Liveness asks whether the process should restart and avoids transient external dependencies.
livenessProbe:
  httpGet: { path: /health/live, port: 8080 }
  periodSeconds: 10

# ② Readiness controls traffic and may include narrow probes for request-critical dependencies.
readinessProbe:
  httpGet: { path: /health/ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3

Alert design

• Prefer user impact, error-budget burn, and sustained backlog over every transient exception.
• Warning creates an investigation window; Critical means delivery or data safety is already affected.
• Include service, environment, safe scope, start time, dashboard, and runbook.
• Join logs and traces with redacted correlation IDs; never put secrets or personal data in labels.
• Detect missing telemetry and a dead alert path with a dead-man signal.

Runbook template

1. Confirm impact: which endpoints, tenants, messages, or jobs are affected?
2. Find the first dependency: readiness, database, RabbitMQ, Redis, or external adapter?
3. Contain risk: remove traffic, pause consumers, fail high-value writes closed, or roll back.
4. Recover facts: after dependency recovery, reconcile outbox, idempotency, task state, and audit.
5. Verify service: run a narrow smoke check and observe error budget plus backlog before reopening fully.
6. Leave evidence: timeline, root cause, customer impact, mitigation, long-term action, and owner.

Drill database loss, RabbitMQ backlog, Redis lock Deny, stopped JobHost, audit write failure, and a wrong production adapter. See observability and disaster recovery.

Current OTel and health semantics

ServiceDefaults registers ASP.NET Core, HttpClient, and Runtime metrics, framework ActivitySources, plus Quartz and database slow-query meters. Traces, metrics, and logs export through OTEL_EXPORTER_OTLP_ENDPOINT.

/health/live selects only the live self check. /health/ready selects ready dependencies. Compatibility endpoint /health runs all checks and must not become a lightweight liveness probe.

# ① Probe process and dependency semantics separately, retaining status and latency.
curl -fsS -o /dev/null -w 'live %{http_code} %{time_total}\n' "$URL/health/live"
curl -fsS -o /dev/null -w 'ready %{http_code} %{time_total}\n' "$URL/health/ready"

# ② Control sampling with standard variables and verify in Staging first.
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.10

Correlate alerts with releases

Dashboards need service.name, service.version, and deployment.environment dimensions. Release markers should link image digest and commit so responders can separate a new version from dependency failure or traffic change.

High-cardinality tenantId, userId, and raw URL IDs do not belong in metric labels. Use controlled logs or traces for tenant investigations.

Validate runbooks

Exercise every Critical alert at least quarterly: inject a controlled failure and verify routing, acknowledgement, containment authority, recovery commands, and evidence capture. Reading a runbook is not execution proof.

Alert → owner acknowledged → impact scoped → traffic/consumer contained
      → dependency recovered → outbox/task/audit reconciled
      → narrow smoke → backlog drains → close with timeline

Completion checklist

• live and ready dependency sets match restart and traffic-removal semantics;
• collector outage cannot silently exhaust application memory or disk;
• alerts cover user impact, error budget, and sustained backlog;
• each alert has owner, dashboard, runbook, and escalation path;
• release, incident, rollback, and recovery correlate through trace context;
• telemetry absence has a Dead Man alert.

First dashboard

Use five rows for API, JobHost, database, messaging, and Redis. Each row shows traffic/throughput, errors, latency, saturation, and the latest release marker. Every panel links to the relevant log or trace query and states timezone plus sampling.

Do not keep attractive metrics without an owner. A panel should support an SLO, capacity decision, release decision, or runbook—or be removed.

Observe at least one full business peak after deployment and compare the prior baseline. Low-volume environments need synthetic smoke and job heartbeats so “no data” is not health.

For CAP, audit, and JobHost, monitor oldest backlog age as well as count. A stable count with rising age often reveals a stopped loop sooner.

• default to the current environment while allowing prior-version comparison;
• store critical queries as reviewed code;
• version dashboard changes with releases;
• use long windows for capacity and short windows for incidents;
• restrict tenant and personal data through dashboard authorization.