HTTP Resilience

BitzOrcas uses IHttpClientFactory for outbound connection management and attaches ExternalRequestLoggingHandler to factory-created clients. The current codebase has no global Polly or AddStandardResilienceHandler pipeline. That boundary matters: existing audit and timeout settings must not be presented as general retry or circuit-breaker support.

What exists today

• Unified lifetime management through AddHttpClient() and typed or named clients.
• ExternalRequestLoggingHandler records method, URL, status, latency, correlation identity, and masked request and response bodies.
• OpenTelemetry HttpClient tracing and metrics instrumentation.
• A dedicated 5-second timeout for Gateway downstream health probes.
• Webhook delivery owns its attempt and retry state machine instead of using a shared HTTP pipeline.

The audit handler reads and rebuilds content streams. Evaluate memory and logging cost before using it for large bodies or streaming responses. Even with masking, avoid auditing complete bodies that serve no operational purpose.

Actual audit-handler semantics

ConfigureHttpClientDefaults covers only factory-created clients. A direct new HttpClient() or connector-owned pipeline is not automatically covered. Integration SDKs may instead bridge their own IRequestLogStore.

The handler calls ReadAsStringAsync on request and response, reconstructs StringContent, and copies content headers. It has no body-size cap, streaming bypass, or content-type allowlist. Large files, binary content, and streams are unsafe fits.

If base.SendAsync throws for DNS, connection, TLS, or timeout, current control flow does not create ExternalRequestRecord; only received responses are recorded. “Every external attempt is audited” is therefore false.

HTTP response received → read/mask body → ExternalRequestRecord
send throws exception  → no record today; exception propagates

Four questions for every integration

1. What is the total budget?

Define how long the caller can wait, then allocate that budget across attempts, backoff, and downstream processing. Propagate cancellation from the entry point to HttpClient; an infinite timeout merely hides a stuck dependency.

2. Which failures are transient?

Candidates commonly include network errors, 408, 429, and selected 5xx responses. Authentication failures, validation errors, and business rejection should not be retried. Leave room in the total budget before honoring Retry-After.

3. Is the operation idempotent?

A nominally safe method may still trigger a downstream side effect. A non-safe method can become retryable under a verified idempotency contract. Judge the dependency contract, not only the HTTP verb.

4. What happens while the circuit is open?

A circuit breaker stops additional pressure; it does not create a business fallback. The caller must choose among returning an error, serving controlled stale data, queuing work, or disabling the dependent capability.

Policy order and budget

Worst-case latency is approximately attempt budget × attempts + backoff and must remain within request timeout or Job shutdown budget.

total budget 8s
├─ attempt 1: 2s
├─ backoff: 200ms
├─ attempt 2: 2s
├─ backoff: 500ms
└─ remaining budget for attempt 3 / response mapping

Retry inside versus outside a circuit breaker changes whether the breaker counts attempts or logical calls. Fix policy order with tests rather than relying on decorator defaults.

Recommended registration shape

Register a separate typed client for each external system. Keep BaseAddress, authentication, timeout, and resilience policy in one composition entry point. Business handlers depend on a domain port rather than composing URLs.

// ① Focus on the contract and control flow; keep validation, cancellation, and typed errors explicit.
services.AddHttpClient<IPricingClient, PricingClient>(client =>
{
    client.BaseAddress = new Uri(options.BaseUrl);
    client.Timeout = TimeSpan.FromSeconds(5);
});

If Polly is introduced, attach integration-specific retry, circuit-breaker, and attempt-timeout policies at that registration point. Split clients or pipelines for idempotent and non-idempotent calls. There is no repository-wide Polly configuration to copy today.

public static class PricingErrors
{
    public static readonly Error Throttled =
        Error.Failure("Pricing.Throttled", "Pricing is temporarily throttled.");
}

// The adapter maps HTTP outcomes to stable integration errors.
public async Task<Result<PriceQuote>> GetQuoteAsync(
    QuoteRequest request,
    CancellationToken cancellationToken)
{
    using var response = await http.PostAsJsonAsync("quotes", request, cancellationToken);
    // A typed 429 is mapped here; transport exceptions remain available to resilience.
    if (response.StatusCode == HttpStatusCode.TooManyRequests)
        return Result.Failure<PriceQuote>(PricingErrors.Throttled);

    response.EnsureSuccessStatusCode();
    return Result.Success((await response.Content.ReadFromJsonAsync<PriceQuote>(cancellationToken))!);
}

Preserve transient exceptions for the resilience layer to decide. Do not turn the first 500 into a business Validation error.

Non-idempotent calls

Payments, sends, and remote creation can be retried only when the provider accepts a stable idempotency key with a documented duplicate contract. Persist the key with the local business fact and reuse it after restart.

A timeout may mean the provider succeeded. Model that as Unknown/Pending, then query or reconcile by remote business key instead of blindly creating again.

Observability and privacy

Record dependency, operation, result category, attempt count, total duration, and circuit state. Mask URL query, Authorization, Cookie, API keys, and bodies. Do not put full URLs or customer identity in metric labels.

For large content, audit metadata, hash, size, and controlled samples. Never buffer a streaming download entirely for audit.

Verification checklist

1. Tests cover DNS failure, connection refusal, timeout, 429, 500, and cancellation.
2. Maximum attempts and wall-clock duration are bounded and cannot create a retry storm.
3. Automatic retry cannot duplicate side effects for non-idempotent requests.
4. Logs and traces correlate attempts without exposing credentials or personal data.
5. Sustained downstream failure has an explicit business fallback and alert.

Also test cancellation during backoff/send/body read, total timeout bounding internal retries, limited half-open probes, bounded Retry-After, and the selected behavior when audit storage fails.

Current gaps

• no global or standard resilience handler;
• no audit body cap or streaming bypass;
• transport exceptions create no ExternalRequestRecord;
• reconstructing StringContent changes concrete content/stream semantics;
• generic body logging carries privacy and memory risk.

These gaps do not prevent integration-specific typed clients, but commercial delivery needs per-connector policies and fault-test evidence.