Webhook network security, rate limits, and production composition

Webhooks lets a tenant-configured URL influence server-side outbound traffic. It is therefore an egress-security boundary. https:// alone is insufficient: DNS, redirects, proxies, ports, cloud metadata, and multi-tenant fairness all matter.

1. Production configuration and fail-closed defaults

Application composition defaults to FailClosedWebhookIpAllowlistPolicy, FailClosedWebhookRateLimitPolicy, and DeliveryLogWebhookDeadLetterQueue. Infrastructure appends production CIDR/Redis policies only when Webhook:Delivery:Enabled=true. If fail-closed remains the resolved implementation, every delivery dies at the guard instead of being silently allowed.

{
  "Webhook": {
    "Delivery": {
      "Enabled": true,
      "RotationOverlapSeconds": 300,
      "IpAllowlist": {
        "Enabled": true,
        "RequireSubscriptionAllowlist": true
      },
      "RateLimit": {
        "Enabled": true,
        "PermitLimit": 60,
        "WindowSeconds": 60,
        "KeyPrefix": "bitzorcas:webhooks:delivery"
      }
    }
  }
}

Redis is registered by the host as a shared IConnectionMultiplexer; it is not configured under this section. KeyPrefix binds through options even though it lacks a [ConfigKey], but automated configuration catalogs may omit it.

2. What readiness proves

webhook-delivery-production has ready/dependency/webhook tags. It requires delivery, IP policy, and rate policy enabled; positive PermitLimit/WindowSeconds; and a connected Redis multiplexer.

It does not probe DNS, egress, certificate trust, DataProtection key ring, CAP consumer, database, HTTP timeout, event-topic alignment, or a receiver. Healthy means the policy baseline is configured—not end-to-end delivery.

# Readiness proves a dependency baseline; it does not replace synthetic delivery.
curl --fail --silent https://api.example/health/ready | jq .

# Confirm production composition did not leave a fail-closed development adapter active.
curl --fail --silent \
  -H "Authorization: Bearer $OPS_TOKEN" \
  https://api.example/api/operations/adapters | jq '.[] | select(.name|test("Webhook"))'

3. Exact CIDR algorithm

An XML comment once says at least one allowed address, but implementation rejects if any resolved address is outside the networks. IPv4-mapped IPv6 normalizes to IPv4; an address without prefix is /32 or /128.

4. Why CIDR is not complete SSRF defense

The check and connection are separate:

1. guard resolves hostname;
2. all results pass CIDR;
3. ordinary HttpClient sends to the hostname;
4. transport can resolve again;
5. default behavior may follow redirects to another URL.

Low-TTL DNS rebinding can change the answer between check and connect. A permitted server can redirect to loopback, link-local, or metadata. A proxy may make the actual destination differ from local DNS.

5. Recommended URL policy

• reject userinfo, fragments, and ports outside an approved set;
• normalize IDN/Punycode before domain policy;
• globally deny loopback, unspecified, multicast, link-local, RFC1918/ULA, and cloud metadata;
• use administrator-approved egress profiles for legitimate private destinations;
• limit DNS answers and resolution time;
• disable redirects or revalidate each bounded hop;
• disable automatic proxies or use only a controlled proxy;
• record final remote IP, TLS protocol, certificate fingerprint, and egress profile—not body.

Tenant-supplied IpAllowlist cannot protect platform internals: a tenant can allow 127.0.0.1/32. Platform global deny ranges must be non-overridable.

6. Current HttpClient boundary

DefaultWebhookHttpClient uses a DI-provided generic HttpClient, sends UTF-8 JSON with headers, reads only status code, and maps HttpRequestException/non-caller TaskCanceledException to network failure.

There is no module-specific handler, timeout, connect timeout, connection cap, ResponseHeadersRead, redirect/proxy policy, minimum TLS, certificate policy, or body limit in source. A generic default timeout is unsuitable for many slow endpoints inside a CAP consumer.

// Webhooks gets a dedicated pool and deadline instead of generic host defaults.
services.AddHttpClient<IWebhookHttpClient, DefaultWebhookHttpClient>(client =>
{
    client.Timeout = TimeSpan.FromSeconds(15);
    client.DefaultRequestHeaders.UserAgent.ParseAdd("BitzOrcas-Webhooks/1");
})
.ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
    // Redirects must return to URL/CIDR policy; the transport must not follow silently.
    AllowAutoRedirect = false,
    UseProxy = false, // Or configure one controlled egress proxy.
    ConnectTimeout = TimeSpan.FromSeconds(3),
    MaxConnectionsPerServer = 20,
    PooledConnectionLifetime = TimeSpan.FromMinutes(2)
});

This example still needs DNS pinning through ConnectCallback or an egress gateway with correct TLS SNI.

7. Redis fixed-window limiter

A Lua script atomically INCRs a key and sets PEXPIRE(window) on the first count:

{prefix}:{tenantId}:{clientId}:{subscriptionId}:{eventType}:{bucket}
bucket = unixSeconds / windowSeconds

Default is 60 per 60 seconds for each subscription/event partition, shared across API instances. Counts continue increasing above the limit until expiry. Disabled policy, absent/disconnected Redis, and script exceptions all fail closed.

Fixed-window boundaries permit close to 2× burst. Raw colon-delimited identifiers reduce readability and do not control Redis Cluster hash slots.

8. Rate limiting currently amplifies operations

The service creates a delivery row and signature before rate policy. A denial immediately becomes DeadLettered, not scheduled after the window. Normal traffic spikes therefore create manual operational debt.

Prefer ingress quota plus durable queue and fair scheduler. A rate decision should set NextAttemptAt rather than make a normal burst terminal. Add global, tenant, target-host, and subscription concurrency limits so one slow receiver cannot exhaust the pool.

9. Metrics and logs

CIDR and Redis policies already log tenant/client/subscription/event plus resolved addresses or quota. ResolvedAddresses exposes customer topology and needs restricted access/retention. Never add URL query, body, secret, or full signature.

10. Release and incident runbook

Before release:

1. readiness and cross-instance Redis contract pass;
2. DataProtection key ring is persisted, shared, backed up, and restore-tested;
3. egress firewall/proxy and subscription CIDR both enforce;
4. dedicated HttpClient timeout/redirect/proxy are fixed;
5. CAP consumer contracts pass;
6. challenge subscription succeeds without a real business effect;
7. dead-letter query/replay/suspend/rotate authorization and audit are drilled.

Redis outage currently dead-letters all new work. Restore Redis, assess backlog, then replay by tenant/event under rate control. For DNS/certificate incidents, suspend affected subscriptions rather than disabling allowlist globally. For key-ring loss, restore the ring or coordinate a new secret; do not treat Unprotect plaintext fallback as recovery.

11. Required network tests

1. IPv4/IPv6/mapped IPv6 and one bad address among many.
2. Empty/NXDOMAIN/timeout/answer change/DNS rebinding.
3. Redirect to public/private/loopback/metadata/HTTP.
4. Userinfo, IDN, fragment, port, oversized URL.
5. Controlled proxy and proxy bypass.
6. Expired/mismatched/unknown-CA TLS and handshake timeout.
7. Redis missing/disconnected/script failure/window edge/cross-instance.
8. Tenant fairness, slow-target exhaustion, and recovery burst.
9. Readiness only when all required adapters are usable.
10. No high-cardinality URL, body, or key leakage.

12. Review commands

# Confirm the final composition and readiness contract together.
rg -n "FailClosedWebhook|AddBitzOrcasProductionWebhookDelivery|HealthCheck" \
  src/Platform/Webhooks src/Hosts -g '*.cs'

rg -n "GetHostAddressesAsync|SendAsync|AllowAutoRedirect|ConnectCallback" \
  src/Platform/Webhooks src/Hosts -g '*.cs'

# The Redis Lua/key contract must retain cross-instance integration evidence.
rg -n "IncrementScript|BuildKey|ScriptEvaluateAsync" \
  src/Platform/Webhooks tests -g '*Webhook*.cs'

Back to Webhooks · subscriptions and authorization · events and GA