Tickets testing, observability, and GA gates

Tickets has useful aggregate and dual-ORM evidence. Its current “Ready” status primarily proves unified persistence and owner-local contract migration, not complete business behavior, cross-module messaging, or production operations. Treat structural readiness and commercial GA as separate gates.

1. Existing test assets

They still do not prove live HTTP for the 36 declarations, actual CAP bytes binding to typed consumers, the Webhook bridge, restricted deep paging, or concurrent conflict behavior.

2. Layered strategy

Unit tests fix pure rules, dual-ORM tests fix storage/query, HTTP tests fix auth/binding/Problem Details, and consumer contracts fix independently deliverable messages. A few E2E paths do not replace deterministic layers.

3. Complete state-machine tests

Enumerate all 6×6 pairs and separately verify same-target idempotency. Every success asserts Status, ModifyTime, ResolvedAt/ClosedAt, event count, and payload; failure changes nothing.

public static IEnumerable<object[]> EveryTransition()
{
    // Expected is an independent business specification, not a call to production rules.
    foreach (var from in AllStatuses)
    foreach (var to in AllStatuses)
        yield return new object[] { from, to, Expected[from, to] };
}

[Theory]
[MemberData(nameof(EveryTransition))]
public void Transition_Should_Match_Spec(
    TicketStatus from, TicketStatus to, bool shouldSucceed)
{
    var ticket = RestoredAt(from);
    var before = Snapshot(ticket);

    // Exercise a public aggregate behavior, not private TransitionTo.
    var result = ApplyPublicBehavior(ticket, to, Now);

    result.IsSuccess.ShouldBe(shouldSucceed);
    if (!shouldSucceed)
        Snapshot(ticket).ShouldBe(before);
}

4. Concurrency red test

Run on real SqlSugar and EF Core with independent scopes. Two writers read the same Version and append comments or change state. Target: one succeeds and one returns Ticket.VersionConflict. Current implementation can let both succeed and erase one change.

// Independent Units of Work read the same persisted version.
var left = await LoadInNewScopeAsync(ticketId);
var right = await LoadInNewScopeAsync(ticketId);
left.Version.ShouldBe(right.Version);

left.AddComment("comment-a", "user-a", "A", Now);
right.AddComment("comment-b", "user-b", "B", Now);

// First commit wins; the stale commit is a typed conflict.
(await SaveAsync(left, expectedVersion: left.Version)).IsSuccess.ShouldBeTrue();
var stale = await SaveAsync(right, expectedVersion: right.Version);
stale.Error.Code.ShouldBe("Ticket.VersionConflict");

// A successful response must never hide an overwritten fact.
var final = await ReloadAsync(ticketId);
final.Comments.Select(x => x.CommentId).ShouldContain("comment-a");

5. Event contract red test

The consumer fixture should reference only the published contract/schema, not Tickets Application/Infrastructure. Generate the envelope through the real producer, then feed the actual Reporting deserializer.

// Execute real Open handler + UoW + outbox and capture broker output.
var opened = await api.OpenTicketAsync(
    new("Cannot login", "MFA fails", "High", DueAt: Now.AddHours(4)));
var message = await broker.TakeAsync("ticket.opened", cancellationToken);

// Wire bytes bind through the actual Reporting typed contract.
var contract = serializer.Deserialize<TicketOpenedIntegrationEvent>(message.Body);
contract.EventId.ShouldNotBeNullOrWhiteSpace();
contract.TicketId.ShouldBe(opened.TicketId);
contract.TicketId.ShouldNotBe("0");

// Tenant protects projection isolation; the current contract lacks Version/sequence.
contract.TenantId.ShouldBe("tenant-a");

The generated registry now adds TicketId, TenantId, Subject, and other public business fields to the parameter dictionary, and publication errors roll back the transaction. This end-to-end test remains expected red because the opened domain event can retain id 0 and the repository has no “actual CAP serializer bytes → Reporting DTO” evidence.

6. HTTP contract matrix

Test all 36 GenerateEndpoint declarations by capability group; for QUERY, also test the same-path POST _query fallback:

• anonymous 401, wrong action 403, correct action reaches Handler;
• route ticketId/body binding and unknown/duplicate field policy;
• SmartEnum strings/numbers/invalid values;
• null/blank and 255/256 subject plus oversized Body/Ids;
• exact response DTO without accidental aggregate exposure;
• NotFound/Forbidden/Conflict/Validation Problem Details;
• StandardCommand timeout only where the source declares it;
• target Idempotency-Key, ExpectedVersion, and If-Match;
• rate limit, request size, and trace correlation.

Host OpenTicketRequest and sibling request records only appear in the JSON source context; generated endpoints use Commands as the real contract. Do not test disconnected request types and call that endpoint coverage.

7. Comment and attachment tests

Comments: same id/same payload, same id/different payload, cross-Ticket, concurrent ids, oversized body, writes during migration, writes after migration, corrupt JSON, and report reconciliation.

Attachments: missing, cross-tenant, PendingUpload, Deleted/Quarantined, public, owner, elevated permission, unauthorized, same AttachmentId/different FileId, duplicate FileId, participant download after binding, and later file deletion.

After Comments cutover add an architecture gate: Tickets Application must not call Ticket.AddComment, and ordinary repository updates must not write CommentsJson.

8. Notification and audit tests

Notification creation Failure now makes the Ticket Command fail, and the Transaction pipeline rolls back. Fix that contract in tests, including actor=requester, actor=assignee, requester=assignee, group expansion, and reassignment dedup. Channel delivery still requires Notifications outbox/attempt evidence, not a mocked service.

Production composition already replaces Null with PersistentTicketAuditSink, and readiness rejects Null. Still prove one audit per write use case, no duplicate on idempotent replay, batch-command policy, rollback on audit exception, and redacted logging.

9. Database and performance baseline

Add TicketId as a stable ordering tail and bound deep paging or use cursors. Move large JSON out rather than increasing timeouts indefinitely.

10. Metrics and logs

Suggested metrics:

• ticket_command_total{command,result,error_code};
• ticket_state_transition_total{from,to,result};
• ticket_version_conflict_total{command};
• ticket_query_duration_seconds{scope,filter};
• ticket_json_bytes{field};
• ticket_event_outbox_age_seconds{topic};
• ticket_projection_lag_seconds{consumer};
• ticket_audit_append_total{result};
• ticket_notification_intent_total{code,result}.

Minimal logs: TicketId, tenant_hash, actor_hash, command, from/to, version, error_code, traceId. Never log description, comment body, StorageKey, notification body, or raw tenant/user ids.

11. Example SLOs

Targets need product/deployment approval: create/state P95 under 300ms without waiting for external delivery; list P95 under 500ms; oldest outbox under 60s; audit append 99.99%; projection P95 lag under 30s; zero lost updates; zero cross-tenant reads.

“Notification sent” must use provider-accepted/delivered indicators, not Ticket Handler success.

12. Incident runbook

Projection stalled: pause the harmful consumer, inspect contract/topic, retain outbox/dead messages, repair, and replay by EventId+Version. Do not manually patch a mart without replay evidence.

Missing comments: freeze writes for that Ticket, compare audit, backup, CommentsJson, and SysComment; identify concurrency overwrite versus migration split-write; recover and add regression coverage.

Attachment exposure: revoke FileAsset/signed URLs, inspect Ticket/FileId/actor evidence, verify whether binding was mistaken for shared authorization, rotate credentials, and repair resource policy.

NullAuditSink in production: treat as compliance incident. Stop high-value writes or enter an explicit degraded Profile; a Warning with green health is insufficient.

13. GA blockers

• HTTP contracts pass for 36 endpoint declarations and every QUERY fallback;
• typed event and Reporting/Webhook wire contracts pass;
• opened event uses final TicketId;
• Start/Reopen can rebuild state through events;
• Ticket update compares/increments Version;
• Comments has one write source;
• participants download files through an explicit policy;
• assignee tenant/existence/active/deleted-state validation;
• assignee skill, capacity, schedule, and invalidation/reassignment policy;
• production audit adapter and Null readiness guard;
• batch event/notification/audit behavior is equivalent to single-item commands or explicitly different;
• SLA claims match policy/jobs/metrics;
• search, deep-page, and large-JSON baselines;
• backup restore and event replay drill.

14. Global review commands

# Producer and consumer must share exact topic and versioned payload.
rg -n "ticket\.(opened|assigned|started|resolved|closed|reopened)|tickets\.opened|Ticket.*IntegrationEvent|CapSubscribe" src tests -g '*.cs'

# Audit replacement, notification failure, and concurrency versioning need direct evidence.
rg -n "NullTicketAuditSink|NotificationService|UpdateWhereAsync|VersionConflict|BuildParameters" src/Platform/Tickets src/Framework -g '*.cs'

# Comments cutover and FileAsset policy remain visible.
rg -n "AddComment|CommentsJson|MigrateTicketComments|EnsureCanAttach|FileAssetOwnerPolicy" src/Platform/Tickets src/Platform/Comments src/Platform/Files -g '*.cs'

# Route contract tests must hit each declaration rather than substitute aggregate tests.
rg -n "\[GenerateEndpoint|/api/tickets|/api/query-options/ticket" src/Platform/Tickets tests -g '*.cs'

Back to Tickets · Lifecycle and SLA · Events and audit