Performance baselines and regression decisions

A performance baseline is a protocol for comparing one scenario under reproducible conditions, not a speed leaderboard detached from hardware, data volume, and version. BitzOrcas can already record engineering-gate durations through its acceptance script. That is not a complete API/database performance gate; runtime scenarios still require stable load, budgets, storage, and regression rules.

Measurement loop

Verify correctness before speed. Missing tenant filters, skipped message delivery, or swallowed errors can improve numbers while being functional regressions.

Two distinct baselines

An “Application.Tests duration” is not “application throughput.” Gate time depends on SDK cache, NuGet, generators, and test count; runtime results depend on data, concurrency, network, and dependency capacity.

Engineering-gate timing

# Produce the 13-step report with status, duration, command, and summary.
scripts/build/report-acceptance.sh

# Preserve environment identity for comparisons using the same SDK/RID class.
dotnet --info > .verify-output/dotnet-info.txt
git rev-parse HEAD > .verify-output/commit.txt

Watch Restore, Build, Unit, Application, Architecture, CodeGeneration, Template, Integration without Docker, and Trim Publish. If one step changes materially, repeat it at least three times before blaming code; cold caches and shared CI noise are common.

# Repeat the suspect boundary; do not add --no-build and change the experiment.
time dotnet test tests/BitzOrcas.Application.Tests --configuration Release
time dotnet test tests/BitzOrcas.Application.Tests --configuration Release
time dotnet test tests/BitzOrcas.Application.Tests --configuration Release

First restore, initial template instantiation, and Docker image pull are cold-start samples and should not be mixed into a warm median.

The first built-in runtime baseline

One reusable template already exists: PortRepositoryParityTests.Authorization_Uncached_Rbac_Should_Emit_Repeatable_Capacity_Baseline (dedicated shard authorization-capacity) drives 10 tenants x 1,000 uncached RBAC reads and emits P50/P95/P99, throughput (ops/s), CPU time, GC allocation bytes, and working-set delta. Reuse its measurement structure and report fields for your next scenario instead of inventing a new format.

Runtime scenario catalog

A commercial framework should cover at least these end-to-end scenarios:

Each scenario needs fixed input and correctness assertions. A list benchmark must validate tenant, total, ordering, and paging—not only HTTP 200.

Baseline record shape

# Example record shape; store real results in the team's traceable artifact system.
scenario: workflow-complete-task
commit: 0123456789abcdef
runtime: .NET 10.0.x
environment: perf-sqlserver-2026-07
dataset: 100-tenants-1m-instances
load:
  concurrency: 32
  duration: 10m
result:
  # Latency, throughput, and errors must come from the same load run.
  throughput_per_second: 0       # Replace with measurement; zero is no framework promise.
  p95_ms: 0
  p99_ms: 0
  error_rate: 0
notes: warm-cache, no deployment overlap

Record commit, runtime, Release/Debug, CPU/memory/OS, container limits, database version, data size, tenant count, index state, cache state, concurrency, request mix, duration, errors, throughput, p50/p95/p99, CPU, allocations, and dependency latency.

Comparability rules

1. Compare only the same hardware class, dependency versions, dataset, and load model.
2. Fix client/server time sources and confirm the load generator is not saturated.
3. Warm JIT, connection pools, and common caches while retaining a separate cold-start scenario.
4. Use multiple samples and report median plus variance, not the fastest run.
5. Record deployment, backup, index maintenance, and other interference.
6. Create a new baseline ID when the method changes; do not splice old curves.

Absolute milliseconds across machines are rarely comparable. Prefer relative change within one environment and use explicit SLO/resource budgets as the final boundary.

Decide regression

Means hide tail latency, while one p99 can be dominated by infrastructure noise. Evaluate throughput, p50/p95/p99, error/timeout rate, CPU, memory/GC allocation, database and broker latency, and queue backlog together.

Example decision policy:
- Correctness assertions must pass 100%; otherwise fail immediately.
- p95/p99 and error rate must remain within scenario budgets.
- Relative regression must repeat across several runs and exceed historical noise.
- Resource-cost growth needs explanation even when latency is unchanged.

Those thresholds are team policy for a specific environment, not universal numbers built into current BitzOrcas source. Do not invent platform-wide TPS or p95 promises in documentation.

Locate a regression

Minimize to one scenario and segment the path: client queue, ASP.NET Core, authorization/tenant pipeline, serialization, ORM/SQL, Redis, RabbitMQ, object storage, or JobHost. Trace the longest span, profile CPU versus allocation, and inspect database plans for scans, indexes, and locks.

If a security guard has a justified cost, record risk benefit, budget impact, owner, and optimization plan rather than silently widening the threshold. Exceptions need an expiry and runtime alert.

Common traps

• Establishing release baselines with Debug or an attached debugger.
• Mixing first JIT/image pull with stable samples.
• Measuring tenant queries without checking isolation.
• Testing success only, omitting timeout, cancellation, conflict, and degradation.
• Watching API latency while shifting backlog to Outbox/JobHost.
• Blocking merges on one noisy shared-CI duration.
• Reusing a baseline ID after data, indexes, or load changed.

Delivery checklist

• Scenario, data, correctness assertions, and SLO are reproducible.
• Raw and summarized results bind commit and environment.
• Cold/warm state, repetitions, and variance are recorded.
• API, database, broker, cache, and JobHost signals correlate.
• Regression evidence includes minimum reproduction, cause, before/after.
• Exceptions have owner, expiry, monitoring, and exit condition.
• Documentation distinguishes existing automation from proposed benchmarking.

See also

• Monitoring and alerting
• Running tests
• Troubleshooting
• Background jobs