Testing Strategy

BitzOrcas is an application template, a package-based framework, and a commercial delivery product. Its tests must answer more than “does this method return the right value?” They must also prove module boundaries, ORM parity, template consumption, and release provenance.

Five layers of evidence

Lower-level tests cannot replace higher-level evidence. Green unit tests do not prove that EF Core and SqlSugar preserve the same concurrency behavior. A green local Consumer contract does not prove that production packages were signed and uploaded to the controlled feed.

Main test projects

The repository now has more than the five projects described in early docs. Common entry points are:

Use tests/ and BitzOrcas.Modern.slnx as the complete inventory.

Match tests to the change

Common commands

# Architecture boundaries
dotnet test tests/BitzOrcas.Architecture.Tests --configuration Release

# Non-Docker integration tests
dotnet test tests/BitzOrcas.Integration.Tests \
  --configuration Release \
  --filter "Category!=Docker"

# Local PackageReference Consumer contract
dotnet test tests/BitzOrcas.Framework.ConsumerContract.Tests \
  --configuration Release \
  --blame-hang-timeout 12m

# Full 14-step local merge gate
scripts/build/verify-all.sh

The docker-integration-contracts workflow splits Docker contracts into 13 shards. To reproduce a failure locally, copy the exact --filter from that job instead of starting with the entire Docker suite.

Consumer Contract scope

The local Consumer Contract packs candidate packages from current source, creates a temporary feed, and isolates NUGET_PACKAGES, HTTP cache, plugin cache, and DOTNET_CLI_HOME. An external project then runs:

restore → build → test → Release publish (currently `PublishTrimmed=false`) → package asset checks

This proves PackageReference closure, generator assets, and source-exposure rules. A separate Commercial GA run must still verify the production feed, signatures, and release evidence.

The local Consumer Contract does not prove trimming or Native AOT safety. Product verify-all.sh runs PublishTrimmed=true for the full API Host, but that does not prove every physically trimmed Consumer Profile. See Consumer Contracts for the gap and completion path.

Test ownership and duplication boundaries

Copying dozens of identical assertions into every layer increases maintenance without independent evidence. High-level tests should cross real boundaries; low-level tests should enumerate business branches.

Pre-merge evidence record

For every verifiable slice, record the exact command, configuration, filter, pass/fail/skip counts, whether Docker or external dependencies participated, and omitted gates with reasons. “Tests passed” cannot distinguish the non-Docker fast set, all 13 shards, and formal GA.

For external or long-running gates, “not run/external prerequisite unavailable” is valid evidence language. A historical result or lower test layer must not be promoted into the current conclusion.

Reading a failure

1. Find the first failed step; later errors may be cascading symptoms.
2. Reproduce with the same configuration, RID, and filter.
3. For Docker failures, separate image/container/readiness problems from business assertions.
4. For parity failures, compare observable semantics rather than internal SQL or implementation shape.
5. For Consumer failures, start with temporary NuGet.Config, Package Source Mapping, and the first missing package.
6. After the fix, run the focused regression and then the full gate required by the slice.

See also

• Running tests
• Architecture tests
• Integration tests
• Performance baselines
• CI/CD
• Consumer contracts

Chapter Navigation

• 16
  01/07

  Architecture tests

  Enforce dependency direction, module seams, generators, commercial delivery, and deletion gates through assemblies, projects, source contracts, and manifests.

  Read
• 16
  02/07

  Unit tests

  Write fast, deterministic tests for aggregates, value objects, Result errors, request rules, and application pipelines as readable business contracts.

  Read
• 16
  03/07

  Integration tests

  Separate API Shell and Docker contracts while proving ORM parity, migrations, messaging, tenant isolation, and real-infrastructure semantics.

  Read
• 16
  04/07

  Fixtures and Seed Data

  Choose pure objects, WebApplicationFactory, Testcontainers, and module-owned CSV seeds for deterministic, isolated, replayable test data.

  Read
• 16
  05/07

  Running tests

  Select unit, architecture, container-free, Docker, template, package-consumption, and acceptance entries by risk while retaining reproducible evidence.

  Read
• 16
  06/07

  Writing new tests

  Start from risk and evidence, put regressions in the correct project, and write tests that fail first, reproduce reliably, remain maintainable, and enter the owning gate.

  Read
• 16
  07/07

  Performance baselines and regression decisions

  Build repeatable performance evidence for engineering gates, APIs, databases, messaging, and jobs while separating existing timing reports from runtime benchmarks still to be established.

  Read