Reporting Testing, Observability, and GA Gates

Reporting is not accepted merely because an endpoint returns 200. Its results must be explainable, traceable, and recoverable. The current code has a query, authorization, export, and six-event projection skeleton, but production trust is still limited by message identity, ordering, the missing Audit producer, and recovery gaps.

1. Current evidence boundary

Existing tests prove that:

• both queries use AuthorizationAction.Read, pass the trusted TenantId to IReportingMartStore, and return a complete PagedResult;
• non-Tenant DataScope, PageSize 1001, and a window longer than 366 days fail before the store runs;
• the runtime Feature evaluator maps the reporting module to platform.reporting;
• export accepts only csv/excel, selects a server-owned builder, requires Tenant DataScope, and rechecks Ticket visibility between output batches;
• TicketReportingEventConsumerTests constructs all six consumer DTOs and covers Opened→Assigned→Started→Resolved→Closed→Reopened, a duplicate Reopened, and an older Closed event;
• architecture tests enforce ORM neutrality, Query Shape use, and generated metadata; the API Shell fails closed when persistence is unavailable.

They do not prove that:

• bytes written by a real Ticket transaction and Outbox bind to all six Reporting DTOs through CAP;
• the Opened payload contains the persisted TicketId; Ticket.Open currently creates an aggregate with "0" and immediately calls Raise(new TicketOpened(ticket.Id,...)) instead of RaiseWhenPersistenceIdAssigned;
• A→B→A replay, equal-time events, concurrent Upserts, and non-adjacent duplicates cannot regress state;
• SqlSugar and EF Core agree on width, ordering, unique conflicts, and total/page consistency;
• any production code writes the Audit daily Mart;
• a real HTTP host has passed the complete 401/403/Feature/DataScope/export matrix;
• a Mart watermark, reconciliation, rebuild, cutover, and rollback path exists.

2. Layered acceptance model

Direct DTO tests are useful for the consumer state machine. They do not replace real Outbox payloads, CAP binding, relational constraints, or a recovery drill.

3. Message bytes and persisted identity

All six Ticket domain events implement IIntegrationEvent and declare [IntegrationTopic]. Generated code maps their public business properties into the payload. DomainEventDispatchPipelineBehavior adds eventId and eventType, then writes the CAP Outbox before transaction commit. Publication failure propagates.

The immediate defect is the timing of the Opened event:

var ticket = new Ticket("0", tenantId, requesterId, subject, priority, nowUtc);

// The repository has not assigned the persisted ID, so this event carries TicketId "0".
ticket.Raise(new TicketOpened(ticket.Id, ticket.TenantId, requesterId, subject, priority.Name, nowUtc));

Assigned, Started, Resolved, Closed, and Reopened run after persistence and use the real TicketId. Reporting can therefore create (TenantId, "0"), then retry every later event because no predecessor exists under the real ID. The fix should use the entity base class’s RaiseWhenPersistenceIdAssigned and an integration test that compares the Outbox TicketId with the repository-assigned ID.

[Fact]
public async Task TicketOpened_OutboxPayload_ShouldCarryPersistedTicketId()
{
    // Exercise the real Command, transaction, and persistence-id assignment path.
    var opened = await OpenTicketThroughRealCommandAndTransactionAsync();
    var message = await ReadOutboxMessageAsync("ticket.opened");
    // Bind the Outbox body with production CAP options to cover the wire contract.
    var bound = BindWithCapOptions<TicketOpenedIntegrationEvent>(message);

    bound.TicketId.Should().Be(opened.TicketId);
    bound.TicketId.Should().NotBe("0");
    bound.EventId.Should().NotBeNullOrWhiteSpace();
}

For every topic, pin the exact name, field names, nullability, timestamp format, enum encoding, missing-field behavior, and compatibility policy. Use a new topic version or explicit schema version for breaking changes.

4. Ordering and idempotency tests

The current consumer rules are precise but limited:

• LastEventId recognizes only a duplicate identical to the latest event;
• except for Opened, OccurredAt < LastUpdatedAt is stale; equal timestamps can still overwrite;
• a non-Opened event without a predecessor throws; store failures are also rethrown so CAP can retry;
• Reopened sets StatusName to Reopened and clears ClosedAt;
• Opened does not compare LastUpdatedAt, and Store Upsert has no version condition.

Keep the existing six-event lifecycle test and add these sequences:

A stable design needs AggregateVersion in messages, LastAppliedVersion in the Mart, and atomic compare-and-set writes. An Inbox or unique consumption record handles non-adjacent duplicates. OccurredAt is diagnostic context, not a business version.

5. Dual-ORM store contract

Run one relational behavior suite against SqlSugar and EF Core. Cover at least:

1. insert/update/query within one tenant and the same TicketId across tenants;
2. Reopened clearing ClosedAt with null;
3. stable Ticket (OpenedAt,TicketId) and Audit (ActivityDate,UserId,ActionType) ordering;
4. PageSize 1..1000, the default maximum offset 100000, and inclusive 366-day boundaries;
5. a 64-character requester/assignee entering a 36-character Mart column;
6. DateTimeOffset/DateTime precision, offsets, and database collation;
7. concurrent first-insert conflict recovery and old-version compare-and-set;
8. allowed differences between EF count+page and SqlSugar ToPageListAsync during concurrent writes;
9. export take 5000 by default, maximum 100000, and maximum offset 10000000;
10. replay after archival or erasure not restoring removed sensitive data.

An in-memory store cannot expose width, ordering, timestamp precision, unique constraints, or isolation behavior and is not a substitute for this suite.

6. HTTP and authorization acceptance

Exercise this matrix against a real host:

Query permissions now match the Catalog’s .read actions, and the Feature evaluator maps reporting -> platform.reporting. The remaining work is composition testing with the production authorization pipeline and generated routes. Keep one current naming inconsistency visible: the central runtime Feature is platform.reporting, while the Reporting owner catalog also declares a disabled reporting.mart; they are not the same switch.

7. Freshness and observability

Current DTOs omit LastUpdatedAt/asOf, and the Mart has no AggregateVersion or consumer checkpoint. Before GA, add at least:

Logs should include TenantId, topic, EventId, TicketId, AggregateVersion once available, outcome, and trace/correlation id. Do not log the full Subject. Return asOf/watermark in responses or a companion endpoint so callers can distinguish a real zero from lagging data.

8. Rebuild and rollback

The repository has no Reporting event archive, checkpoint, rebuild CLI, or reconciliation job. The following is a GA target, not a current capability:

1. pin schema/consumer versions and record tenant, operator, reason, and starting watermark;
2. create a shadow Mart with production constraints and indexes;
3. replay a traceable source by tenant, aggregate, and version;
4. capture live delta and catch up to the cutover watermark;
5. compare counts, status distribution, invariants, samples, and orphan rows;
6. atomically switch readers and writers while watching errors, lag, and authorization denial;
7. keep the old table for an approved rollback window, then apply retention policy.

9. Incident response

CAP repeatedly retries a missing predecessor

Compare the TicketId in Opened with later events. If Opened is "0" and later events use the real ID, the aggregate raised the event before persistence identity assignment. Do not only insert a Mart row manually. Fix event timing, then replay affected Tickets in order from a trusted Outbox/archive.

Projection state regresses

Compare every EventId, OccurredAt, and source business order. The consumer has no AggregateVersion, and equal timestamps can overwrite. Stop incorrect writes for the affected partition, introduce version gating and CAS, then rebuild those Tickets. A direct database edit will recur on replay.

Audit daily does not advance

Production source has no caller of UpsertAuditActivityDailyAsync. Restarting a consumer cannot restore a producer that does not exist. Implement the aggregator, checkpoint, timezone/date normalization, and idempotent backfill before adding lag alerts.

A release causes widespread 403

Check platform.reporting, the exact .read/.export grants, authorization cache, and final DataScope in that order. reporting.mart and platform.reporting are separate definitions; do not diagnose the owner-catalog switch as the runtime module switch.

10. GA gates

The first three actions are to raise Opened with the persisted TicketId, test real Outbox bytes for all six topics, and add AggregateVersion plus Inbox/CAS and observable outcomes. Then implement the Audit daily producer and rebuild path.

11. Review commands

# Inspect current query, export, consumer, and Feature evidence.
rg -n "Reporting|TicketReportingEventConsumer|platform.reporting" tests -g '*.cs'

# Locate placeholder identity, delayed-ID events, and consumer retry/order rules.
rg -n 'new Ticket\(|RaiseWhenPersistenceIdAssigned|ticket.Raise|LastEventId|LastUpdatedAt|will retry' \
  src/Platform/Tickets src/Platform/Reporting src/Framework -g '*.cs'

# Find the Audit daily production writer; current production source should show only the port and Store.
rg -n "UpsertAuditActivityDailyAsync" src -g '*.cs'

Back to Reporting · Ticket projection · Audit daily · Mart storage