AIManage testing, operations, and commercial GA gate

AIManage has runnable persistence and provider-call scaffolding, but every request crosses identity, secret, external-network, nondeterministic-output, and billing boundaries. Commercial GA evidence must be stronger than ordinary CRUD evidence.

1. Existing automation

Missing are real provider contracts, first-byte timing, cancellation races against provider cancellation, key corruption/rotation, file-skill authorization, content safety, capacity, SLOs, and recovery.

2. Minimum regression set

# Run application behavior and adapter contracts first.
dotnet test tests/BitzOrcas.Application.Tests/BitzOrcas.Application.Tests.csproj \
  --filter "FullyQualifiedName~AIManage|FullyQualifiedName~SemanticKernelChatClientFactory|FullyQualifiedName~RagServiceAdapter|FullyQualifiedName~AgentServiceAdapter"

dotnet test tests/BitzOrcas.Architecture.Tests/BitzOrcas.Architecture.Tests.csproj \
  --filter "FullyQualifiedName~AIManage"

# Dual-ORM parity requires the repository's Docker-backed integration environment.
dotnet test tests/BitzOrcas.Integration.Tests/BitzOrcas.Integration.Tests.csproj \
  --filter "FullyQualifiedName~AIManageStores_Should_Behave_The_Same_On_SqlSugar_And_EfCore"

A release candidate also needs HTTP contract tests against an isolated provider stub, pinning authentication, Problem Details, NDJSON, timeout, cancellation, usage, redaction, and idempotency.

3. Threat matrix

4. Idempotency and consistency

// Both concurrent requests carry the same stable RequestId.
var command = new SendMessage.Command(conversationId, "turn-42", content, modelId);
var results = await Task.WhenAll(
    sender.Send(command, cancellationToken).AsTask(),
    sender.Send(command, cancellationToken).AsTask());

results.ShouldAllBe(result => result.IsSuccess);
// Pin both the external charge boundary and the durable message result.
await provider.Received(1).CompleteAsync(Arg.Any<ChatRequest>(), Arg.Any<CancellationToken>());
await store.ShouldContainOneUserAndOneAssistantAsync("turn-42", cancellationToken);

The RequestId is required and the IAIMessageRequestCoordinator claim state machine is enforced. Also cover: provider success followed by assistant-save failure, crash after user save, partial stream persistence, duplicate completion (replay without provider call), concurrent counters/default providers, cache construction races, and usage-ledger replay.

5. Provider failure injection

Simulate DNS, TLS/certificate failure, 401/403, missing model, 429 with Retry-After, connect timeout, first-byte timeout, midstream interruption, invalid/huge frames, missing usage, empty output, filtered finish, slow stream, pool exhaustion, and recovery.

Error mapping must hide endpoint, key, raw provider body, and stack. Retry decisions depend on typed error, idempotent attempt, and budget. Do not silently switch providers after partial output.

6. Streaming acceptance

Test time, not only values. If a stub emits at 0/200/400ms, the client must receive near those boundaries rather than all at 400ms. Disconnect tests assert database state, provider cancellation, absence of leaked work, and a recoverable client contract.

Validate proxy buffering, compression, idle timeout, and maximum duration for every supported deployment topology. X-Accel-Buffering: no is not universal proof.

7. Capacity and cost

Cover 1/10/50/500 history records, 1KB/32KB input, 1KB/100KB/1MB output, 1/10/100 concurrent streams, 1/8/32/64 cache keys, and slow long streams. Record P50/P95/P99, first byte, total duration, allocation, GC, connections, provider QPS, tokens, cost, and database trips.

Budget tests cover tenant day/month caps, user/workspace rate, per-turn limits, concurrency, anomaly spikes, and billing reconciliation.

8. Telemetry

Minimum telemetry includes request outcomes by workspace/provider/model/error, first-byte/full/gap duration, input/output tokens and cost, provider status/429/timeout/circuit state, client-cache activity, partial/cancel/orphan turns, skill reload/parse/denial, and RAG search/index lag.

Logs may include tenant, workspace, conversation, turn, provider, model, error, and correlation IDs. They must not contain API keys, prompts, responses, retrieved chunks, or unclassified filenames. Content forensics belongs in a controlled audit store.

9. Key and recovery drill

Backups include aggregates/child tables, messages, future usage ledger, and the Data Protection key ring. Recovery proves old credentials decrypt, old/new keys coexist during rotation, a wrong key ring produces a red health state rather than outbound ciphertext, counters reconcile, failed attempts recover, and vector indexes can rebuild.

10. Suggested SLOs

Separate platform admission from provider quality. Example: 99.9% admission decisions within 100ms; within provider SLA, 99% streams receive first frame within 2s; 99.9% completed turns reconcile message and usage within one minute. Segment by model, region, and response size.

11. GA blockers

Delivered: conversation ownership (tenant + user), idempotent turns (RequestId + IAIMessageRequestCoordinator), incremental streaming with typed kind frames, and stable non-leaking stream error contracts. Remaining blockers:

1. Conversations can reference missing/inactive workspaces.
2. Arbitrary models and fixed options bypass model metadata.
3. Decryption falls back to the raw string and key lifecycle is undefined.
4. Default-provider, model ownership, and cache concurrency are unproven.
5. File-skill routes are authentication-only and skills/RAG/Agent are overclaimable.
6. Classification, redaction, safety, injection defenses, and retention are absent.
7. Token, cost, budget, quota, and reconciliation are absent.
8. Provider health, resilience, fallback evidence, dashboards, and alerts are absent.
9. HTTP, security, provider, streaming timing, capacity, and recovery evidence is absent.

P0 delivers workspace validation, model allowlists, and credential fail-closed behavior. P1 adds usage/budget, safety, provider resilience, file-skill authorization, telemetry, and recovery. RAG/tools follow only after per-invocation authorization and audit; multi-agent orchestration is later product scope.

12. Release evidence package

The release artifact should link the threat-model review, HTTP/provider contracts, dual-ORM reports, first-byte timing trace, concurrency and failure runs, key-rotation and restore drill, cost reconciliation, content-safety evaluation, dashboard screenshots, alert tests, and signed operational runbook. A green unit-test run alone is not GA evidence.

13. Rollback criteria

Stop or roll back when cross-tenant/owner access is observed, credentials cannot decrypt consistently, usage cannot reconcile, first-byte or error rates breach the agreed window, provider fallback violates residency, unsafe output controls fail, or operators cannot identify and contain a failed attempt. Disable affected workspace/provider capability before attempting broad retries.

rg -n "GetConversationAsync\(|CancellationToken.None|return cipherKey|DisableRequestTimeout" \
  src/Platform/AIManage src/Hosts/BitzOrcas.Api -g '*.cs'

rg -n "TODO|FIXME|// \.\.\." src/Platform/AIManage tests -g '*.cs'

AIManage overview · Skills, RAG, and Agent · Streaming and usage