Notifications testing, operations, and GA gates

Notification success is not a 200 from Create. The intended user must receive correct, unique, safe, traceable content through allowed channels, and failures must be explainable, retryable, and reconcilable. The source has useful aggregate/persistence foundations but not yet the delivery/template loop needed for that SLO.

1. Current test assets

The repository contains:

• NotificationAggregate state and restoration tests;
• NotificationPreference and ReadMarker unit tests;
• notification/template read-query handler tests;
• TemplateSecurityChecker, VariableAnalyzer, NamingConverter, and StorageProjector tests;
• NotificationArchivePagination tests;
• dual-ORM parity for Notification/Template repositories and read models;
• Notifications Infrastructure architecture and production-adapter readiness tests;
• Workflow notification adapter/dispatcher tests;
• template-migration tenant-backfill parity.

There is no direct coverage of NotificationService.CreateAsync, NotificationCreatedConsumer, MultiChannelDeliveryAdapter, NotificationOrchestrator, TemplateAppService, TemplateRendererService, ScribanTemplateCompiler, the 19 HTTP contracts, or provider failure recovery.

2. Test layers

Unit tests fix pure rules, contract tests fix boundary/failure semantics, and E2E uses deterministic provider fakes for one complete delivery. A real-provider sandbox smoke does not replace deterministic contracts.

3. Create/delivery red test

[Fact]
public async Task Create_Should_Persist_Delivery_Fact_When_Inbox_Is_Disabled()
{
    // The user hides Inbox but permits email; a logical fact must remain reloadable.
    preferences.Set("tenant-a", "user-1", "security.login", new(
        InboxEnabled: false,
        ExternalDeliveryEnabled: true));

    var created = await service.CreateAsync(
        "tenant-a", "user-1", "security.login",
        "New device sign-in", "A new device signed in",
        severity: NotificationSeverity.Warning,
        metadataJson: "{\"email\":\"user@example.com\"}");

    created.IsSuccess.ShouldBeTrue();
    // The current implementation did not save it, so this is an intentional red assertion.
    (await repository.FindByIdAsync(created.Value!.NotificationId, default))
        .IsSuccess.ShouldBeTrue();
}

Also prove atomic Save+outbox, one row per BusinessId, target-tenant consumer reload, one attempt per channel, and retry/dead state instead of swallowed failures.

4. Template red test

[Fact]
public async Task Render_Should_Resolve_Tenant_Template_And_Reject_Unsafe_Content()
{
    // A tenant-owned template exposes the current empty-TenantId lookup defect.
    await templates.SeedAsync(
        tenantId: "tenant-a",
        key: "Tickets.Assigned.Email.zh-CN",
        body: "<script>alert(1)</script>{{ user_name }}");

    var validation = await renderer.ValidateTemplateAsync(
        "title", "<script>alert(1)</script>{{ user_name }}",
        TemplateEngine.Scriban,
        new Dictionary<string, object> { ["user_name"] = "Alice" });

    validation.Value!.IsValid.ShouldBeFalse();
    // The target facade carries tenant; current RenderAsync hard-codes an empty tenant.
    var rendered = await facade.RenderAsync(
        "tenant-a", "Tickets.Assigned.Email.zh-CN", ModuleCaller.Notification,
        new Dictionary<string, object> { ["user_name"] = "Alice" });
    rendered.IsFailure.ShouldBeTrue();
}

Version tests must include 1.0.9/1.0.10/1.0.11, concurrent updates, Description, locale uniqueness, import/export fidelity, write-time safety validation, and Preview report preservation.

5. HTTP and authorization contracts

For every generated route, verify 401, wrong-permission 403, correct-permission handler entry, route parameters, JSON, Problem Details, timeout, and rate-limit policy. High-risk drift:

• POST /api/notifications asks for Create, but notification.create is absent;
• Activate uses template.update, not template.activate;
• Render/Preview/Validate use template.view while template.preview is unused;
• current-user APIs cannot choose recipient;
• Read/Unread/Archive return the same NotFound for another user’s id.

Do not merely inspect attributes. Exercise the real catalog seed and authorization middleware.

6. Observability

Business metrics

• notification_intent_total / deduplicated_total;
• inbox_projection_total / hidden_by_preference;
• delivery_attempt_total{channel,result,provider};
• delivery_latency, queue_age, retry_count, dead_count;
• template_resolve{scope,locale,channel,result};
• render_latency, compile_cache_hit, validation_reject, degraded_total;
• unread_count and inbox-query latency;
• hot/archive rows, archive lag, and reconciliation_delta.

Minimal log fields

EventId, NotificationId, tenant_hash, code, channel, attempt, provider, error_code, and traceId. Never log Body, template variables, MetadataJson, Email, Phone, or raw UserId. Sanitize vendor exceptions, which often contain recipient data.

7. SLOs and alerts

Example targets require product approval:

Alerts must derive from attempt state and queue age, not log searches alone. The absence of an attempt ledger prevents reliable measurement and is a GA blocker.

8. Incident runbook

Without persisted attempts, “replay failed notifications” is unsafe. Never bulk-call providers from log entries; the original result may have been unknown rather than failed.

9. Capacity and recovery

Test intent throughput, a hot tenant, 100,000 unread for one user, MarkAllRead, deep hot/archive pages, 10/100/10,000 templates, 1/100/1,000 variables, body size, SMS segments, provider timeout, and CAP backlog.

Backup/restore must cover SysNotification, Archive, Preference, all three template tables, and CAP outbox/inbox—plus DeliveryAttempt after it exists. Reconcile by id, business key, version, and hashes, and prove delivered attempts are not resent.

SQL Server notification archival is provider-specific. EF Core read parity or non-SQL Server operation does not prove physical archival; publish an explicit support matrix.

10. GA blockers

• Create permission matches catalog; Activate/Preview use dedicated permissions or remove misleading codes.
• HTTP/background paths use one EffectiveTenant and explicit RequireUserId semantics.
• NotificationIntent has a business idempotency key and atomic Save+outbox.
• Inbox=false does not break external delivery; mandatory-notice policy exists.
• Consumer establishes tenant context, does not swallow failure, and supports retry/dead/replay.
• Per-channel DeliveryAttempt has uniqueness, status, ProviderMessageId, and reconciliation.
• EnabledChannels works and contact PII is not retained indefinitely in Inbox metadata.
• Template resolution supports tenant/PLATFORM/locale/channel.
• Only implemented engines are accepted; CallerModule is server-authorized.
• Create/Update require compile, schema, security review, and publication approval.
• Preview preserves validation; render failures never return raw template source.
• HTML uses context encoding and a mature allowlist sanitizer.
• Numeric versioning, uniqueness, concurrency, and Description are fixed.
• Bilingual seeds, 100+ export paging, and import/export fidelity are fixed.
• HTTP, dual-ORM, CAP, provider, capacity, archive, and restore suites pass.

11. Verification commands

# Current module test assets.
dotnet test tests/BitzOrcas.Unit.Tests/BitzOrcas.Unit.Tests.csproj \
  --filter 'FullyQualifiedName~Notification|FullyQualifiedName~Template'
dotnet test tests/BitzOrcas.Application.Tests/BitzOrcas.Application.Tests.csproj \
  --filter 'FullyQualifiedName~Notifications'
dotnet test tests/BitzOrcas.Architecture.Tests/BitzOrcas.Architecture.Tests.csproj \
  --filter 'FullyQualifiedName~NotificationsInfrastructureArchitectureTests'

# Sweep current critical boundaries.
rg -n 'skipExternalDelivery|EnabledChannels|GetByKeyAsync\(\s*string.Empty' \
  src/Platform/Notifications -g '*.cs'
rg -n 'DeliveryAttempt|ProviderMessageId|DeduplicationKey|ModuleAuthorization' \
  src/Platform/Notifications -g '*.cs'

Back to Notifications