Webhook delivery, idempotency, retry, and dead letters

WebhookDeliveryService orchestrates event selection and HTTP sending. Its defining limitation is that one delivery row is both the idempotency record and latest-state snapshot. It is not a delivery-attempt ledger, and no background retry scheduler exists.

1. Exact first-attempt order

An unregistered type fails the entire DeliverAsync. A StartDelivery failure for one subscription is omitted by the outer loop: only successful logs are appended, and the method still returns Success. The CAP consumer ignores the returned Result. A local storage failure can therefore trigger neither CAP retry nor a returned error.

2. Idempotency key and concurrency window

The database unique index and Repository lookup use (EventId, SubscriptionId). New delivery ids are UUID v7 in 32-character N format.

The business key is appropriate, but check-then-insert still races: two instances can both observe absence and Add, leaving the unique index to reject one at commit. Repository does not translate that conflict into a read of the winning row.

// The fast path reduces duplicate writes, but the database unique key is authoritative.
var existing = await deliveries.FindByEventAndSubscriptionAsync(
    tenantId, eventId, subscriptionId, cancellationToken);
if (existing is not null)
    return existing.ToLog();

try
{
    await deliveries.AddAsync(newDelivery, cancellationToken);
    await unitOfWork.SaveChangesAsync(cancellationToken);
    return newDelivery.ToLog();
}
catch (UniqueConstraintException)
{
    // A concurrent instance won; reading its row converts the race to idempotent success.
    return (await deliveries.GetRequiredByKeyAsync(
        tenantId, eventId, subscriptionId, cancellationToken)).ToLog();
}

Current dual-ORM parity invokes StartDelivery twice sequentially, not concurrently.

3. What one delivery row retains

There is no start time, duration, error category/message, NextAttemptAt, response detail, trace id, worker, key id, or per-attempt record. Manual retry overwrites StatusCode, Signature, and OccurredAt, so operators cannot reconstruct four prior failures.

4. HTTP result classification

DefaultWebhookHttpClient returns only (StatusCode, NetworkFailure):

• 2xx → Succeeded;
• network error, internal timeout, null, 408, 429, or 5xx → transient;
• every other status, including visible 3xx and other 4xx → non-transient;
• transient below MaxAttempts → Retrying;
• otherwise → DeadLettered.

HttpClient may follow redirects by default, so policy might never see the original 3xx. Response body and Retry-After are ignored; DNS, connect, TLS, timeout, and reset collapse into one class.

5. Backoff exists without scheduling

Default MaxAttempts=5; after completed attempt N:

delaySeconds = min(300, 2^N × 5)
N=1 => 10s
N=2 => 20s
N=3 => 40s
N=4 => 80s

WebhookRetryDecision returns Status, ShouldRetry, and NextAttemptAt. DeliveryService consumes only Status. There is no persistence or Quartz/JobHost/CAP-delay consumer, so a 503 can leave the row in Retrying indefinitely until an operator calls retry.

platform.webhook.maxRetries=5 and platform.webhook.retryIntervalSeconds=60 are registered settings, but WebhookRetryPolicy reads neither. Its max is a constructor default and its delay is exponential, not 60 seconds.

6. CAP retry is not Webhook retry

CAP can redeliver the platform message. Normal DeliverAsync sees an existing AttemptCount greater than zero and returns without HTTP, preventing duplicate effects. Consequently CAP retry cannot substitute for an HTTP delivery scheduler.

A sound split is: CAP reliably reaches a Webhooks inbox; a delivery scheduler owns per-subscription HTTP attempts; (event, subscription) deduplicates delivery creation; (delivery, attemptNumber) deduplicates workers; receiver EventId inbox deduplicates external effects.

7. Manual retry semantics

POST /api/webhooks/deliveries/{deliveryId}/retry:

1. tenant-scoped lookup of the payload-free log;
2. current subscription lookup (Deleted can no longer be found);
3. Succeeded returns WebhookDelivery.AlreadySucceeded;
4. separate tenant-scoped retrieval of original PayloadJson;
5. same EventId/EventName/body becomes a new WebhookEvent;
6. forceRetry=true bypasses the existing-attempt short circuit;
7. current subscription URL, allowlist, Client state, and current secret are used;
8. the same delivery row increments AttemptCount.

This is not a byte-identical request replay. Body stays the same, but timestamp/signature change. The send uses current subscription.TargetUrl while the public log keeps the original TargetUrl. After a subscription URL update, replay goes to the new address but the log still displays the old address.

public static class WebhookDeliveryErrors
{
    public static readonly Error TargetChanged =
        Error.Conflict(
            "WebhookDelivery.TargetChanged",
            "Choose explicitly between the historical and current destination.");
}

// Delivery has the historical destination; Subscription has the destination used now.
var delivery = await deliveryStore.FindAsync(tenantId, deliveryId, cancellationToken);
var subscription = await subscriptions.FindAsync(
    tenantId, delivery.SubscriptionId, cancellationToken);

if (!StringComparer.Ordinal.Equals(
        delivery.TargetUrl, subscription.TargetUrl.AbsoluteUri))
{
    // Do not let an operation named replay silently change the receiving system.
    return WebhookDeliveryErrors.TargetChanged;
}

return await deliveryService.RetryAsync(tenantId, deliveryId, cancellationToken);

8. Maximum attempts do not cap operator sends

Manual retry rejects only Succeeded. DeadLettered and AttemptCount≥MaxAttempts still send HTTP; policy merely marks another failure DeadLettered. An authorized user can trigger unbounded external side effects.

Separate automatic retry budget, operator replay budget/approval, break-glass replay, original/current endpoint choice, dry-run challenge, and payload retention eligibility. Audit actor, reason, incident, attempt delta, target, result, and correlation id. The Webhooks retry endpoint has no explicit activity audit; the OpsExtension path does, subject to a real production sink.

9. The dead-letter port is not a separate queue

Production composition registers DeliveryLogWebhookDeadLetterQueue. It validates that a log is DeadLettered and returns Task.CompletedTask; status in the delivery table remains the fact. It does not write a broker, object store, or isolation table.

OpsExtension pages non-deleted DeadLettered WebhookDelivery rows by (OccurredAt desc, Id desc). Deleting a dead letter soft-deletes the delivery. The Webhooks query predicate does not explicitly exclude IsDeleted, so visibility then depends on global EntitySet filtering.

10. Completion-write failures are ignored

DeliveryService does not inspect CompleteDeliveryAsync Result. If lookup/update fails, it still builds an in-memory completed log and returns Success; the DLQ port may receive a log not durably marked DeadLettered.

HTTP already happened while its outcome was not recorded. A recoverable model persists AttemptStarted, sends, then completes with ExpectedVersion. An uncertain crash window becomes Unknown, not fake Succeeded.

11. Recommended recoverable model

Add jitter, bounded Retry-After, and fair tenant/client/subscription scheduling. Lease acquisition prevents duplicate workers; (DeliveryId, AttemptNumber) is unique. Encrypt, classify, retain, and erase payload by policy.

12. Test matrix

1. Full 2xx set, redirect on/off, 400/408/409/429/500/503.
2. DNS/connect/TLS/request timeout/caller cancellation.
3. Retry-After seconds/date/bad/oversized and jitter.
4. Two instances concurrently creating one idempotency row.
5. CAP replay no duplicate HTTP; scheduler executes at due time.
6. Historical/current URL, secret rotation, and allowlist change.
7. Completion failure, process crash after HTTP success, Unknown recovery.
8. Manual permission, audit, budget, approval, concurrent clicks.
9. Stable attempt-history paging and no body disclosure.
10. Backpressure, fairness, restart recovery, and dead-letter drill.

13. Review commands

# Currently no consumer uses ShouldRetry or NextAttemptAt.
rg -n "ShouldRetry|NextAttemptAt|WebhookRetryDecision" src -g '*.cs'

rg -n "CompleteDeliveryAsync|EnqueueAsync" \
  src/Platform/Webhooks/BitzOrcas.Platform.Webhooks.Application/WebhookDeliveryService.cs

rg -n "Idempotent|Concurrent|UniqueConstraint|StartDeliveryAsync" \
  tests -g '*Webhook*.cs'

Back to Webhooks · signatures and secrets · events and GA