Running tests

BitzOrcas tests are not one linear suite with different durations. The solution combines pure-logic tests, source/project structure contracts, container-free API contracts, real-infrastructure contracts, generator consumption, template matrices, and a Consumer Contract that packs locally. Select evidence by changed risk and then widen the scope; that is faster and easier to explain than starting with bare dotnet test.

Choose an entry point

Fast local feedback

# Pure domain and application control flow; use Release like the main gates.
dotnet test tests/BitzOrcas.Unit.Tests --configuration Release
dotnet test tests/BitzOrcas.Application.Tests --configuration Release

# Architecture boundaries and API/Host contracts that start no containers.
dotnet test tests/BitzOrcas.Architecture.Tests --configuration Release
dotnet test tests/BitzOrcas.Integration.Tests \
  --configuration Release --filter 'Category!=Docker'

Keep commit, SDK, configuration, and filter stable during one diagnosis. Do not add --no-build unless the same configuration has already built; otherwise stale binaries can run.

Reproduce one failure precisely

Prefer the FullyQualifiedName or matrix filter from CI output. Namespace, class, and method identify intent better than a common method fragment.

# One architecture rule class.
dotnet test tests/BitzOrcas.Architecture.Tests \
  --configuration Release \
  --filter 'FullyQualifiedName~CommercialPackageCatalogTests'

# Website Docker shard; this requires an available Docker daemon.
dotnet test tests/BitzOrcas.Integration.Tests \
  --configuration Release \
  --filter 'Category=Docker&FullyQualifiedName~BitzOrcas.Integration.Tests.Website.' \
  --blame-hang-timeout 10m

List tests first when checking whether a filter is too broad:

# Discover without execution and confirm project plus category.
dotnet test tests/BitzOrcas.Integration.Tests \
  --configuration Release --list-tests \
  --filter 'FullyQualifiedName~Website.'

Generators and templates

“The generator builds inside the repository” and “an isolated project consumes it as a NuGet analyzer” are separate facts. Templates must also prove that Profiles instantiate, restore, and build without referencing product source.

# Verify generation logic, packaged consumption, and template consumption.
dotnet test tests/BitzOrcas.CodeGeneration.Tests --configuration Release
dotnet test tests/BitzOrcas.Generator.Package.Tests --configuration Release
dotnet test tests/BitzOrcas.ConsumerTemplate.Tests --configuration Release

# Run the authoritative canonical template matrix.
scripts/build/verify-template.sh

After changing generator package assets, use the repository’s package preparation flow. A stale package that happens to exist in the global NuGet cache is not evidence.

Consumer Contract

The local Consumer Contract packs on demand, creates isolated feeds/caches, generates a consumer project, and restores, builds, runs tests, and publishes. It currently passes /p:PublishTrimmed=false, so it proves Release publish consumption—not trimmed publish for every Profile.

# Allow a useful hang-dump window; first run creates packages and isolated caches.
dotnet test tests/BitzOrcas.Framework.ConsumerContract.Tests \
  --configuration Release \
  --blame-hang-timeout 12m

Full API Host trim evidence comes from step 12 of verify-all.sh; neither substitutes for the other. Commercial GA also requires the catalog, Profile, and provenance evidence in Consumer contracts.

The fifteen verify-all.sh steps

scripts/build/verify-all.sh is the authoritative local pre-merge entry point. Step numbers match its own log output:

1. Gitleaks secret scanning when installed locally; CI keeps the full scan.
2. Restore BitzOrcas.Modern.slnx.
3. dotnet format --verify-no-changes plus unused-using gates.
4. Release build.
5. Solution-wide non-integration tests (Consumer Contract and Architecture run as serial gates afterwards).
6. Local PackageReference Consumer Contract, serialized.
7. XML file checks.
8. XML comment checks.
9. Architecture process profiles (cold and warm) plus the full architecture suite including shallow-module deletion gates.
10. Category!=Docker integration tests.
11. Per-RID restore and a PublishTrimmed=true publish of the full API Host.
12. Roslyn semantic reflection gate (ADR 0102/0103).
13. No-T-SQL gate (ADR 0033).
14. Production deployment and portability asset checks.
15. OpenAPI artifact plus platform-sdk client drift gate.

# Run at repository root; the script exits on the first hard failure.
scripts/build/verify-all.sh

Docker contracts sit outside those fifteen steps and run in the thirteen shards of the docker-integration-contracts workflow. “Local verify-all passed” therefore does not mean every Docker provider contract passed.

Generate a citable acceptance report

report-acceptance.sh writes step, status, duration, command, and summary to .verify-output/acceptance-report.md. It currently has thirteen report steps and is not identical to verify-all.sh: for example it runs Unit and Application separately and its container-free Integration step excludes the API namespace. Do not call it a logging wrapper around the fourteen-step script.

# Generate a Markdown summary and detailed per-step logs; do not commit them.
scripts/build/report-acceptance.sh

# Read the first FAIL and reproduce with its original command and environment.
sed -n '1,220p' .verify-output/acceptance-report.md

PR evidence should include commit SHA, SDK/RID, command, result, duration, and omitted gates. “Passed locally” is not reproducible evidence.

Handle failures

1. Start with the first failed step; later failures may be missing-artifact cascades.
2. Reproduce with the original configuration, RID, filter, and environment.
3. For restore, inspect isolated feed, NuGet.Config, and cache path; do not switch to an unknown source.
4. For Consumer failures identify pack, restore, build, run, or publish stage.
5. For Docker inspect daemon, container logs, and readiness before assertions.
6. For hangs retain blame evidence and inspect cancellation plus unbounded waits.
7. After fixing, run the regression first and then the complete owning gate.

Evidence checklist

• Use authoritative BitzOrcas.Modern.slnx, not a legacy solution.
• Record whether Docker ran and which of the eleven shards were covered.
• Record whether Consumer publish was trimmed; never infer it from Release.
• Logs expose no secret, license material, or connection string.
• Temporary feed, cache, containers, and .verify-output stay out of commits.
• Record global sweep commands and expected zero-result old patterns.

See also

• Testing strategy
• Integration tests
• Architecture tests
• CI/CD