Auditing testing and production operations

The acceptance standard is not “a row exists.” Evidence must be formed at the correct boundary, remain tenant-safe and nonsecret, fail visibly, preserve category semantics through query and retention, and let an operator identify the affected time, instance, and record range.

1. Current automated evidence

Audit tests are distributed across four layers:

Structure tests prove that types, tokens, or registrations exist; they do not prove runtime semantics. Cross-provider ordering, real database rollback, process crashes, and high-volume pruning need integration or process fixtures.

2. Suggested commands

Run from the BitzOrcasVNext root with the repository-pinned SDK:

# General audit, persistence, and outbound-HTTP unit tests.
dotnet test tests/BitzOrcas.Unit.Tests/BitzOrcas.Unit.Tests.csproj --filter "FullyQualifiedName~Auditing|FullyQualifiedName~Audit|FullyQualifiedName~ExternalRequestLoggingHandler"

# Alert, export, retention-policy, and SIEM application tests.
dotnet test tests/BitzOrcas.Application.Tests/BitzOrcas.Application.Tests.csproj --filter "FullyQualifiedName~Auditing"

# SQL query, retention HTTP surface, and CAP Outbox integration evidence.
dotnet test tests/BitzOrcas.Integration.Tests/BitzOrcas.Integration.Tests.csproj --filter "FullyQualifiedName~Audit|FullyQualifiedName~CapOutbox"

# Dependency direction and adapter contracts.
dotnet test tests/BitzOrcas.Architecture.Tests/BitzOrcas.Architecture.Tests.csproj --filter "FullyQualifiedName~Audit"

Before relying on an OR filter, use dotnet test --list-tests to confirm actual matches. A command that selects zero tests is not a green gate.

3. Producer matrix

Every source needs success, business failure, system exception, cancellation, and context coverage:

4. Alert-rule operations

The alert API includes:

• GET/PUT /api/audit/alerts for list and save;
• DELETE /api/audit/alerts/{ruleId} with optimistic version soft deletion;
• POST /api/audit/alerts/{ruleId}/toggle;
• POST /api/audit/alerts/{ruleId}/dry-run, count only;
• GET /api/audit/alerts/history for trigger and notification outcomes.

Rules accept only controlled Category, Success/Failure result, threshold 1–10,000, window 1–1,440 minutes, cooldown 0–1,440 minutes, and semicolon-separated recipient identifiers. Notification interprets recipients; rules hold no URL or credential.

Every minute AuditAlertEvaluationJobExecutor enumerates tenants, counts events in the window, publishes audit.alert.triggered after threshold and cooldown checks, and stores notification success/failure history.

Alert save also uses auditing.audit.view. As with retention policy, restrict this permission to audit administrators until management actions are split.

5. SIEM management and delivery

Management API

Read uses operations.siem.view; configuration, replay, drop, and test use Update. Management changes attempt an Activity audit, but that best-effort record does not block a saved configuration.

Controlled profiles

An optional TLS CA is resolved by CredentialName through ISecretStore; a missing secret skips the round rather than downgrading to cleartext. Configuration bounds BatchSize to 1–1,000, MaxRetryAttempts to 0–100, validates the port, and uses a version to prevent lost updates.

Runtime semantics

The SIEM job enumerates tenants every 30 seconds. Each round first retries at most 100 due DLQ entries, then queries configured categories, formats CEF, and sends. A failed new event enters DLQ with an initial 30-second delay; later failures use exponential backoff capped at 256 seconds and eventually become Dropped. Test events affect neither watermark nor DLQ.

Category configuration currently validates only nonblank text, not membership in AuditCategory. If every configured value is unknown, forwarding returns Success with no sends. The UI should use catalog values only, and the API should reject unknown values.

SiemForwardJobExecutor also accumulates failed tenants but returns Success. Operators must inspect tenant metrics, LastError, watermark stagnation, and DLQ rather than relying on Quartz state alone.

6. Existing signals and gaps

Available signals include:

• ChannelAuditQueue.RejectedCount, alerted on the first and every 1,000th rejection;
• writer Debug batch latency, Error retry attempt, and first/every-tenth failure alert;
• persisted SIEM Sent/Failed/Retried/Dropped, watermark, LastError, and UpdatedAt;
• alert trigger history and Notification state;
• retention Tenant/Scope/Days/Deleted structured logs plus Activity/Exception evidence.

The general pipeline has no dedicated System.Diagnostics.Metrics meter and exposes no supported Queue Depth, Oldest Age, Enqueue Rate, or Persist p95. Add low-cardinality instruments before documenting concrete metric names. TenantId, UserId, CorrelationId, Path, and free-form Module do not belong in metric labels.

7. On-call thresholds

8. Diagnosis flow

Before remediation, preserve instance, deployment version, time range, Category, Tenant, and last known AuditId. Do not immediately run broad pruning, DLQ drop, or duplicate business commands; those actions can obscure the original failure boundary.

9. Pre-release fault drills

1. fill a Capacity-1 Channel and verify async wait, sync rejection, sampled alert;
2. fail the Store repeatedly and recover it, proving same-batch retry, no conflicting duplicate, and released upstream backpressure;
3. create an unknown SqlSugar commit and Mongo cross-collection partial completion, proving stable-ID repair;
4. run SIGTERM, expired grace period, and SIGKILL separately and document provable/unprovable windows;
5. verify pre-enqueue redaction with JSON secret, Bearer, URI password, card number, and oversized payload;
6. compare seven categories, equal timestamps, second page, and cursor across SqlSugar/Mongo/Dapper;
7. prune two tenants by every scope, preserving permanent classes and the other tenant;
8. verify alert ResultCondition with separate Success and Failure fixtures;
9. execute SIEM twice with a new event between runs, then exercise missing TLS secret, DLQ retry, replay, and drop;
10. fail Notification, Secret Store, tenant directory, and policy store and ensure job state reports failure honestly.

10. GA gates

At shift handoff, preserve aggregate SIEM metrics first and export actionable deliveries separately. Record both capture times in the incident:

# Capture watermark, failure, retry, and drop counters first.
curl -fsS -H "Authorization: Bearer <TOKEN>" \
  https://<HOST>/api/operations/siem/metrics > siem-metrics.json
# Preserve delivery rows that need an explicit replay/drop decision.
curl -fsS -H "Authorization: Bearer <TOKEN>" \
  https://<HOST>/api/operations/siem/deliveries > siem-deliveries.json

• no silent drop in the general queue; retry, idempotency, shutdown, and forced-termination boundaries are in the SLO;
• high-value entity changes use the CAP transactional Outbox and non-CAP fallback has an explicit deployment restriction;
• seven-category fidelity across SqlSugar/Mongo/Dapper, with the SqlSugar TraceId gap closed;
• regression coverage for JSON/form/URI/token/card and length bounds; outbound bodies remain uncollected;
• retention runs by Tenant+Scope, host defaults are floor-gated, and preview labels the Request boundary;
• alert ResultCondition affects query and tenant failure propagates to job result;
• SIEM two-round watermark is green, unknown Category is rejected, and failed tenants fail the job;
• AutoCreateTables and AuditShardingSchedule are either wired or removed from public configuration;
• required SLO signals such as Queue Depth/Oldest Age have supported meters rather than log estimates;
• Production prune, SIEM drop, and cross-tenant investigation have separate permission, approval, purpose, and non-prunable evidence.

Previous: Retention and compliance · Back to Auditing