Auditing queue, batch delivery, and durability

The general audit pipeline now follows a no-silent-drop rule: asynchronous entry points wait for capacity, while synchronous entry points fail explicitly when they cannot wait. This is substantially stronger than the old DropOldest behavior, but it remains an in-process queue rather than a transactional Outbox or disk-backed broker.

1. Channel contract

ChannelAuditQueue is a singleton bounded queue:

• ChannelCapacity defaults to 10,000;
• BoundedChannelFullMode.Wait;
• one reader and multiple writers;
• EnqueueAsync applies backpressure and observes caller cancellation;
• TryEnqueue returns false when saturated or closed, and the dispatcher then throws InvalidOperationException;
• every record receives a stable AuditId and passes normalization, redaction, and length validation before first enqueue.

RejectedCount—also exposed through the compatibility name DroppedCount—counts explicit synchronous rejection, not silent background loss. The first rejection and every 1,000th thereafter sample an alert. Alert delivery is single-flight to prevent the alert itself from recursively filling the audit queue.

2. Batch flush timing

The writer waits for a first record, then gathers until one of these conditions holds:

• BatchSize, default 100;
• BatchIntervalSeconds after the first record, default five seconds;
• Channel completion, which returns a partially gathered batch.

At low volume, aggregation delay is bounded at about five seconds. At high volume, full batches flush sooner. Startup validation covers all three values: Capacity 1–1,000,000, BatchSize 1–10,000 and no larger than Capacity, and Interval 1–3,600 seconds.

3. Retry and idempotency

AuditBatchWriter retains the current batch and does not read another one until the Store succeeds. On failure it:

• retries the same batch indefinitely until success or forced host cancellation;
• uses a one-to-thirty-second bounded backoff for general failures;
• recognizes connection-pool pressure and backs off for 35–90 seconds to return capacity to login and HTTP work;
• raises BatchWriteFailure on the first and every tenth failed attempt;
• never swallows OOM.

Stable AuditId makes full-batch retry verifiably idempotent:

• SqlSugar writes the six table groups in one database transaction. An unknown commit outcome is retried by checking the target split table and verifying that any existing row is identical evidence;
• Mongo uses $setOnInsert upserts per category collection, then rereads and verifies every document. Collections do not share a transaction, but retry fills a partially completed batch;
• identical duplicates under one ID collapse; different evidence under one ID fails instead of overwriting history.

4. Exclusion and suppression

The dispatcher evaluates two policies before enqueue:

• Audit:Excludes filters known high-volume, low-value records by Category and Pattern; up to 1,000 validated rules are allowed;
• AuditEmissionScope explicitly suppresses runtime audit for infrastructure work such as schema initialization and seed.

An intentionally excluded record is neither a rejection nor an overflow. Operations reporting should distinguish policy-based non-capture, synchronous rejection, and the forced-process-loss window.

5. Shutdown and crash semantics

AuditBatchWriter.StopAsync calls ChannelAuditQueue.Complete() first, preventing new writes, and then waits for the reader to consume remaining records and finish its current retries. A normal shutdown can therefore:

1. close the write side;
2. return an incomplete final batch;
3. persist each batch before exit;
4. cancel retry only when the host’s shutdown budget expires.

This is not durable queueing. General audit records still may be lost during:

• SIGKILL, OOM, power loss, or forced container eviction;
• an insufficient grace period while the Store is still unavailable;
• a process crash after evidence is formed but before enqueue;
• cancellation of a waiting EnqueueAsync that the caller chooses to contain.

Each application instance owns its own queue; there is no cross-instance ordering. Evidence that must commit with business state belongs in the entity-audit or business-specific transactional Outbox. WORM, signature, or trusted-timestamp requirements need a dedicated ledger.

6. Choosing a delivery class

Several classes can coexist. A use case must declare its required guarantee before selecting a sink; an interface named Audit does not automatically provide regulatory non-repudiation.

7. Configuration

Audit:
  # Bounded capacity per host process.
  ChannelCapacity: 10000
  # Maximum records in one batch.
  BatchSize: 100
  # Maximum aggregation seconds after the first record.
  BatchIntervalSeconds: 5

A useful first estimate is capacity >= max(0, P - W) × T, where P is peak production rate, W is sustainable Store throughput, and T is the transient outage duration to absorb. Also budget for maximum record size, GC pauses, connection-pool pressure, category round trips, and shutdown drain.

8. Production signals

At minimum, track:

• cumulative synchronous rejection and its sampled alerts;
• Store latency, batch size, failure count, and consecutive retry attempt;
• connection-pool classification and recovery time;
• graceful-drain duration and shutdown-budget exhaustion;
• host restart count plus the last persisted AuditId/OccurredAt before restart;
• SIEM delivery backlog separately from audit-storage backlog.

The source does not currently expose complete Queue Depth or Oldest Age metrics. Add a supported observability port before making an SLO depend on them; log volume is not an exact queue gauge.

9. Fault-injection contract

• Capacity 1/2/N: async waiting, sync rejection, ordering, and cancellation;
• BatchSize/Interval boundaries and a partial batch after Channel completion;
• SqlSugar group failure, unknown commit outcome, rollback failure, and stable-ID conflict;
• Mongo single/cross-collection partial completion, incomplete verification, and evidence conflict;
• ordinary database outage, pool exhaustion, and OOM as distinct paths;
• graceful shutdown, exhausted grace period, SIGTERM, and SIGKILL;
• alert-sink failure, recursion guard, and every-1,000 sampling;
• request latency, memory, and database recovery at 1x/5x/10x peak load.

Previous: Capture and normalization · Next: Storage and query