Health checks

Health checks answer two different questions: is the process alive, and can this instance receive traffic now? A database check in liveness creates restart storms during dependency jitter. A readiness endpoint that always reports Healthy leaves broken instances in the load-balancer pool.

Four endpoints

All four currently allow anonymous requests. Default output reveals neither connection strings nor License payloads, but production ingress should still restrict source, request rate, and visibility.

Liveness

AddServiceDefaults() registers a self check with no external dependencies:

// ① Focus on the contract and control flow; keep validation, cancellation, and typed errors explicit.
services.AddHealthChecks()
    .AddCheck(
        "self",
        () => HealthCheckResult.Healthy("alive"),
        tags: [ServiceDefaultsExtensions.LivenessTag]);

Liveness should succeed while the process can handle requests. Database, RabbitMQ, Redis, and third-party failures should not directly restart the Pod.

Two layers of readiness evidence

The API Host’s base readiness checks configuration consistency and real dependencies. Licensing has a separate license tag and is excluded from ready: before issuance, login, password recovery, navigation, and LicenseManagement remain reachable while normal business messages still fail closed in the license pipeline.

Configuration and DI consistency

RuntimeDependencyReadinessHealthCheck verifies:

• database and RabbitMQ are configured together for CAP Outbox mode;
• a configured Redis has a registered connection multiplexer;
• MinIO mode has endpoint, access key, and secret key;
• enabled Webhook delivery has IP allowlist, limiter settings, and connected Redis limiter.

An API Shell may intentionally omit database, RabbitMQ, and Redis and still report Healthy. Do not treat Shell-mode Healthy as evidence that the full business runtime is ready.

Real connectivity

Persistence mode additionally registers:

• database-connectivity, which opens the provider connection and checks its state;
• rabbitmq-connectivity, which probes the AMQP TCP port with a three-second timeout;
• narrow checks owned by cache, S3, Webhook delivery, and other adapters; licensing is exposed separately at /health/license.

Short database and RabbitMQ failures currently return Degraded. /health/ready maps Degraded to 503, removing the instance from traffic without causing liveness restarts.

Kubernetes example

# ① Replace sample values through controlled configuration; never commit production secrets.
livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10

readinessProbe:
  # ② Readiness failure removes traffic without creating an endless restart loop.
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3

Use a startupProbe for slow startup instead of greatly weakening liveness thresholds. Probe timeout should exceed the intended bound of an individual check but remain short enough to remove a broken instance promptly.

Adding a check

The adapter that owns a dependency should own its health check:

[RegisterHealthCheck("search-connectivity", "ready", "search")]
public sealed class SearchConnectivityHealthCheck : IHealthCheck
{
    // ① Registration metadata adds this narrow probe to the generated ready/search catalog.
    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        // ② Execute a narrow, bounded probe with no sensitive output—not a full business query.
    }
}

The API Host currently scans loaded BitzOrcas* assemblies for [RegisterHealthCheck] at the composition root. Conditional database and RabbitMQ checks remain explicitly registered to preserve runtime-mode semantics. This is a bounded Host reflection path; do not copy it into Framework or business modules.

A check should:

• use cancellation and a short timeout;
• never migrate schema, seed, write, or repair;
• avoid passwords, connection strings, tokens, and internal exception details;
• keep a stable name and explicit tags;
• distinguish invalid configuration, temporary unreachability, and bad business data.

Common mistakes

• Readiness contains only self and is therefore always Healthy.
• Liveness queries the database, so a network blink restarts every instance.
• Production starts in API Shell mode but calls its Healthy result business readiness.
• A check runs an expensive business query or external write.
• Every instance probes a struggling dependency too aggressively and worsens the incident.

Release acceptance

Capture live, ready, license, and aggregate-health evidence. In a controlled environment, block database, RabbitMQ, Redis, or object-storage access and prove readiness removes traffic while liveness remains healthy. Restore the dependency and prove the replica re-enters service automatically.

Also prove that an unavailable License leaves /health/ready available while /health/license and aggregate /health return 503, normal business operations are denied, Shell mode is not accepted as production readiness, and response bodies reveal no connection or License details. A healthy-state screenshot cannot prove failure semantics.

Related

• Observability
• Deployment
• GA gate