Webhook event wiring, operations, testing, and GA

The most dangerous false completion is having producer classes, consumer classes, topic constants, and tests without any proof that a real business transaction reaches CAP with the exact wire contract the consumer expects and then reaches an HTTP receiver.

1. Five sources of truth currently disagree

A deliverable event crosses:

1. the business module’s actual publisher/topic;
2. Webhooks EventSubscriptionCatalog governance;
3. [CapSubscribe] on the runtime consumer;
4. IWebhookEventTypeRegistry;
5. external EventType plus required Client Scope.

Both “should equal” links drift today.

2. Source audit matrix

WebhookEventTypes says its values are shared CAP topics, subscription event names, and external payload names. Source does not uphold that promise.

3. The CAP consumer discards typed failure

WebhookPlatformEventConsumer.DeliverAsync awaits the service but does not inspect its Result. An unregistered event or typed repository error does not throw, so CAP considers the message complete. DeliveryService also drops per-subscription failure from its returned list.

// The entrypoint fixes the topic; the envelope carries that topic's versioned fact.
private async Task DeliverAsync(
    string eventType,
    PlatformWebhookEventEnvelope envelope,
    CancellationToken cancellationToken)
{
    var result = await deliveryService.DeliverAsync(
        new WebhookEvent(
            envelope.EventId,
            envelope.TenantId,
            eventType,
            envelope.PayloadJson,
            envelope.OccurredAt),
        cancellationToken);

    if (result.IsFailure)
    {
        // Throw transient infrastructure failure only; isolate and alert contract failure.
        // Production must distinguish terminal contract/configuration from transient storage.
        throw new WebhookEventConsumptionException(result.Error);
    }
}

Do not throw every failure forever. EventTypeNotRegistered is terminal configuration; temporary database loss is retryable. Stable error classes and alerts are required.

4. Choose one versioned envelope

The event owner should define a typed integration event with an explicit versioned topic. Webhooks should consume that exact contract, or a dedicated adapter should convert it to an external envelope. Do not make each module handcraft JSON strings.

// An explicit topic survives namespace/type refactors.
[IntegrationTopic("platform.files.finalized.v1")]
public sealed record FileFinalizedV1(
    string EventId,
    string TenantId,
    string FileId,
    string FileName,
    string ContentType,
    long Size,
    string ContentHash,
    // OccurredAt is business-fact time, not CAP-consume or HTTP-send time.
    DateTimeOffset OccurredAt) : IIntegrationEvent;

{
  "specVersion": "1.0",
  "eventId": "019c48cb7ba47000953d5af574ddc531",
  "eventType": "platform.files.finalized.v1",
  "occurredAt": "2026-07-15T08:00:00Z",
  "tenantId": "tenant-a",
  "data": {
    "fileId": "file-42",
    "fileName": "contract.pdf",
    "contentType": "application/pdf",
    "size": 482193,
    "contentHash": "sha256:..."
  }
}

The owner approves data classification. Do not serialize full internal aggregates: TenantId disclosure, personal filenames, and Ticket descriptions require explicit policy.

5. Independent consumer contract tests

Do not import producer constants to build expected topic; both sides can then change incorrectly and pass. Freeze:

• literal topic;
• content type/serializer options;
• required envelope fields/types;
• EventId/TenantId/OccurredAt semantics;
• data schema and compatibility rules;
• Webhooks registry plus required scope;
• HMAC fixed vector;
• real producer-to-fake-receiver result.

[Fact]
public async Task File_finalized_v1_should_reach_external_receiver()
{
    // The consumer owns this literal; importing the producer constant hides shared mistakes.
    const string expectedTopic = "platform.files.finalized.v1";

    await files.FinalizeAsync(fileId, cancellationToken);
    var capMessage = await capOutbox.WaitForAsync(expectedTopic, cancellationToken);
    capMessage.Json.ShouldContain("\"eventId\"");
    capMessage.Json.ShouldContain("\"tenantId\"");

    var request = await receiver.WaitForEventAsync(cancellationToken);
    // Verify both propagated topic and a real signature made with the one-time secret.
    request.Headers["X-BitzOrcas-Event-Type"].ShouldBe(expectedTopic);
    receiver.VerifySignature(request, oneTimeSecret).ShouldBeTrue();
}

6. Schema evolution

Add optional fields within v1; removal, rename, meaning/unit changes, or closed-enum changes generally require v2. Consumers ignore unknown fields. Producers dual-publish or provide a migration period before sunset. If v1/v2 share EventId, receiver deduplication must include EventType/version.

The current singleton Dictionary registry overwrites duplicate event names, with no conflict detection, version metadata, or owner. Generate an immutable registry from compile-time governance and validate producer/subscriber closure at startup/build time.

7. OpsExtension dead-letter path

OpsExtension pages DeadLettered rows for the current tenant, invokes real manual replay, soft-deletes dead letters, displays/retries CAP failed messages, and writes IActivityAuditSink after replay.

Webhooks also exposes its own retry endpoint without explicit activity audit. Different permissions and audit semantics create an approval bypass. Commercial delivery should select one operator path and separate integration administrators from platform on-call.

8. Existing evidence

Present:

• WebhookSignatureTests: determinism, secret/body tampering, Verify;
• WebhookPlatformTests: basic classification, secret-safe summary, sequential idempotency, Scope denial, original-body replay, Succeeded replay denial, strict JSON, tenant denial;
• WebhookProductionPolicyTests: fail-closed, basic CIDR, missing Redis, DLQ port;
• WebhookProductionAdapterContractTests: real Redis shared state and readiness;
• WebhookProductionOperationsDrillTests: missing config, CIDR rejection, Redis-failure logs;
• dual-ORM parity for subscription/delivery/body/tenant isolation;
• architecture tests for ORM-neutral Infrastructure, unified models, and read stores.

Missing: proof that any Files/Billing/Tickets/Workflow event is consumed; real HTTP receiver; nine endpoint contracts; automatic retry; redirect/rebinding; signature after repeated Save; overlap rotation; concurrent idempotency.

9. Observability and SLO

Measure three stages separately:

1. producer transaction → CAP outbox committed;
2. CAP published → delivery scheduled;
3. delivery due → receiver 2xx.

One success rate cannot distinguish disconnected topics, CAP backlog, guard denial, and customer endpoint failure. A tiered SLO might require 99.9% of deliverable events to succeed or reach actionable dead letter within five minutes. Permanent customer 400 need not count as platform outage but remains a delivery outcome.

Trace producer, CAP message, EventId, DeliveryId, AttemptId, and HTTP client span. Current delivery rows have no trace/correlation or attempt history.

10. GA gates

Contract

• every public event has owner, version, topic, schema, scope, PII class, and sunset policy;
• compile-time producer/governance/consumer/registry closure is green;
• independent tests do not share incorrect producer constants;
• all nine HTTP OpenAPI/Problem Details contracts are frozen.

Reliability

• durable scheduler, lease, jitter, Retry-After, and attempt history;
• concurrent idempotency, crash windows, completion failure, and restart recovery;
• bounded/approved/audited batch replay;
• payload encryption, retention, and erasure.

Security

• DNS pinning/egress proxy, redirects, global deny ranges, ports, timeout, and body size;
• DataProtection backup/restore and double-encryption regression;
• versioned KeyId rotation and secret-exposure runbook;
• actual runtime webhooks.delivery entitlement.

Operations

• readiness + synthetic delivery + adapter report release gate;
• producer/CAP/delivery dashboards and alerts;
• Redis, DNS, certificate, customer 429/5xx, and key-ring drills;
• on-call diagnosis without raw database, secret, or payload access.

11. Global review commands

# Inspect producer, governance, and consumer together—not only the Webhooks tree.
rg -n "CapSubscribe|IntegrationTopic|EventDefinition|files\.finalized|ticket[s]?\.opened|invoice-issued" \
  src/Platform/Files src/Platform/PlatformBilling src/Platform/Tickets \
  src/Platform/Workflow src/Platform/Webhooks -g '*.cs'

rg -n "DeliverAsync\(|IsFailure|GetValueOrThrow|logs.Add" \
  src/Platform/Webhooks/BitzOrcas.Platform.Webhooks.{Application,Infrastructure} -g '*.cs'

rg -n "new PlatformWebhookEventEnvelope|PlatformWebhookEventEnvelope" src tests -g '*.cs'

# Missing scenario names expose unfinished endpoint, concurrency, rotation, and network evidence.
rg -n "Webhook.*(Endpoint|Consumer|Receiver|Concurrent|Rotation|Redirect|Rebind)" tests -g '*.cs'

Back to Webhooks · delivery and retry · OpsExtension · Files · Tickets