Multitenancy testing and production operations

Multitenancy defects rarely crash immediately. They return plausible data from the wrong tenant, contaminate a cache, or give an operations identity a platform-wide data surface. Acceptance must prove that crossing the boundary is impossible, not merely that a TenantId was resolved.

1. What current tests prove

A green test means code matches the assertion. A test that encodes platform-wide Host bypass is not GA security evidence.

2. Minimum isolation fixture

Create two tenants with the same business key for every provider. Different identifiers can make a broken query accidentally return one row and hide a missing tenant predicate.

public async Task SeedIsolationFixtureAsync(IOrderStore store)
{
    // The duplicated OrderNumber forces TenantId to be the isolation predicate.
    await store.InsertSystemAsync(new Order("order-a", "tenant-a", "SO-001"));
    await store.InsertSystemAsync(new Order("order-b", "tenant-b", "SO-001"));

    // A deleted row covers include-soft-deleted repository paths as well.
    var deleted = new Order("order-a-deleted", "tenant-a", "SO-DELETED");
    deleted.Delete(actorId: 1001);
    await store.InsertSystemAsync(deleted);
}

Then prove that tenant-a cannot read tenant-b, include-soft-deleted remains isolated, update and delete predicates carry TenantId, and context reuse never retains the wrong tenant.

3. Request-entry matrix

Host operate-as and Debug do not currently satisfy every target cell. Keep those rows as failing gates until the implementation is repaired.

4. One contract for both ORM providers

[Theory]
[MemberData(nameof(PersistenceProviders))]
public async Task Normal_store_must_never_cross_effective_tenant(string provider)
{
    // Each provider gets a clean database and identical tenant-a / tenant-b fixtures.
    await using var fixture = await TenantDatabaseFixture.CreateAsync(provider);
    await fixture.SeedSameBusinessKeysAsync();

    using var userScope = fixture.PushUser(AuthenticatedUser("tenant-a"));
    using var tenantScope = fixture.PushTenant(new CurrentTenant("tenant-a"));

    // Normal and include-deleted paths must preserve the same tenant boundary.
    var active = await fixture.Orders.ListAsync(includeSoftDeleted: false);
    var all = await fixture.Orders.ListAsync(includeSoftDeleted: true);

    active.Should().OnlyContain(x => x.TenantId == "tenant-a");
    all.Should().OnlyContain(x => x.TenantId == "tenant-a");
}

Add a Host operate-as contract with the same fixture. It should fail today because both providers bypass for CallerType.Host.

5. Formal impersonation contract

An end-to-end impersonation test must cover the session, not only IssueAsync:

1. issue endpoint checks permission and step-up;
2. grant owner matches operator tenant and target is serviceable;
3. signed claims reconstruct the complete CurrentTenant tuple;
4. business data uses EffectiveTenantId and audit retains ActorTenantId;
5. expiry or grant deletion rejects the next request with 401;
6. Scope limits endpoints and data; MaxSessions limits concurrency;
7. cache, files, search, and reports use the same effective tenant.

var token = await impersonation.IssueAsync(
    operatorTenantId: "operations",
    operatorUserId: "1001",
    targetTenantId: "tenant-b",
    cancellationToken);

// The first request sees tenant-b and records grant plus original-tenant facts.
var first = await api.GetOrdersAsync(token.Value!.AccessToken, cancellationToken);
first.Rows.Should().OnlyContain(x => x.TenantId == "tenant-b");

await grants.DeleteAsync(token.Value.GrantId, cancellationToken);

// Middleware checks grant state per request, so the old token is immediately denied.
var revoked = await api.GetOrdersAsync(token.Value.AccessToken, cancellationToken);
revoked.StatusCode.Should().Be(HttpStatusCode.Unauthorized);
revoked.ErrorCode.Should().Be("TenantImpersonation.Revoked");

This is the target contract. The repository does not yet expose the complete issuing and claim-reconstruction path needed to run it.

6. Cache, file, and job tests

Use the same DocumentId, FileName, Setting Key, and message business key in both tenants:

• cache: tenant-b cannot hit tenant-a’s document; impersonation switches key ownership;
• file: every server-stored object key includes a tenant segment and cross-tenant signing fails;
• settings: tenant override wins over global fallback and mutation invalidates derived caches;
• consumer: a message TenantId conflicting with authoritative ownership is rejected;
• workflow timer: callback CurrentTenant, persistence context, and filter are equal;
• audit retention: runs per tenant and does not silently use "0".

7. Production observability

Every request and job should emit structured fields:

Never log a complete JWT, secret, or raw sensitive header. Create separate security events for operate-as, Debug, guard bypass, and platform-wide queries.

8. Troubleshooting path

Capture TraceId, caller type, effective and actor tenants, plus safe SQL or cache-key summaries before isolating traffic. Do not disable tenant filtering merely to check whether a row exists.

9. Failure drills

Before GA, rehearse:

1. stale host-map cache after a domain change;
2. tenant suspension between two requests;
3. grant deletion, token expiry, and clock skew;
4. consumer retry after tenant deactivation;
5. Redis loss and cache refill without boundary loss;
6. equivalent contracts after switching EF Core and SqlSugar;
7. explicit permission, approval, audit, and rate limits on global operations.

10. Source and test review

# Run the existing tenant-focused unit tests to detect accidental behavior drift.
dotnet test tests/BitzOrcas.Unit.Tests/BitzOrcas.Unit.Tests.csproj \
  --filter "FullyQualifiedName~Tenant|FullyQualifiedName~Tenancy"

# Every bypass should have a documented purpose and contract test.
rg -n "IgnoreQueryFilters|ClearAndBackup|IsTenantFilterBypass|isHostBypass" \
  src/Framework src/Platform src/Hosts -g '*.cs'

# Review default fallbacks and every explicit tenant scope.
rg -n 'TenantId.*\?\?.*"0"|DefaultTenantId|PushTenant|BeginScope' \
  src/Framework src/Platform src/Hosts -g '*.cs'

11. GA gate

• Every request-entry row has success and rejection tests.
• EF Core, SqlSugar, cache, files, search, reports, and messaging pass same-key A/B isolation.
• Host operate-as sees only the target; global queries use a dedicated audited port.
• Formal impersonation closes issuance, reconstruction, scope, step-up, sessions, revocation, and audit.
• JobHost uses one CurrentTenant track across business and persistence services.
• The runbook can trace tenant source, data surface, cache key, and grant from a TraceId.

Back: Background work and resources · Back to Multitenancy