Files testing, operations, and GA gates

Files acceptance cannot stop at uploading and downloading one small file. Delivery evidence must cross client, API, database, object storage, and JobHost. It must prove that untrusted bytes cannot become a business asset early, tenant and owner boundaries hold, and the two sources of truth converge after failure.

1. Current test evidence

There is no S3-compatible behavior contract, orphan-cleanup regression test, HTTP end-to-end file test, malicious-file test, quota test, or load test.

2. Unit state matrix

Cover every source state, action, and stable error:

[Theory]
[InlineData("PendingUpload", "mark",     true,  "Uploaded")]
[InlineData("PendingUpload", "finalize", false, "PendingUpload")]
[InlineData("Uploaded",      "finalize", true,  "Finalized")]
[InlineData("Finalized",     "finalize", false, "Finalized")]
[InlineData("Finalized",     "delete",   true,  "Deleted")]
[InlineData("Deleted",       "delete",   false, "Deleted")]
public void State_transition_should_be_explicit(
    string from,
    string action,
    bool expectedSuccess,
    string expectedState)
{
    // Start every row from an explicit state; the fixture must not hide a transition.
    var state = new FileAssetState(FileAssetStatus.FromName(from));

    // The switch forces every new action to enter the decision table.
    var result = action switch
    {
        "mark" => state.MarkUploaded(),
        "finalize" => state.Finalize(),
        "delete" => state.Delete(),
        _ => throw new ArgumentOutOfRangeException(nameof(action))
    };

    result.IsSuccess.ShouldBe(expectedSuccess);
    state.Status.ShouldBe(FileAssetStatus.FromName(expectedState));
}

Test the aggregate separately because it permits Pending to advance twice in one Finalize, unlike the state object.

3. Metadata and hostile input

Cover at least:

• Size 0, negative, 1, maximum allowed, and overflow edges;
• 36/37-character FileId, 50/51 OwnerType, 255/256 FileName;
• blank, mixed-case, parameterized, and forged ContentType;
• blank, non-hex, wrong-length, mixed-case, and mismatched SHA-256;
• multipart-shaped ETag;
• ../, CRLF, quotes, Unicode composition, and bidi file names;
• cross-tenant storage prefixes and similar prefix tenant-a2;
• empty objects, HEAD during upload, eventual consistency, and overwrite races.

The aggregate validates neither hash format nor a platform size ceiling. Freeze current behavior first, then drive a contract upgrade with target tests.

4. Provider contract suite

Run the same suite against Local, S3-compatible, and each connector provider:

public abstract class FileStorageContract
{
    // Derived suites replace provider construction only; behavioral assertions stay identical.
    protected abstract IFileStorage CreateStorage();

    [Fact]
    public async Task Upload_metadata_read_and_delete_should_converge()
    {
        // Use a tenant-scoped unique key so concurrent suites never share an object.
        var storage = CreateStorage();
        var key = $"tenants/contract/{Guid.CreateVersion7():N}";
        var bytes = "contract-body"u8.ToArray();

        await storage.UploadAsync(
            key,
            new MemoryStream(bytes),
            "text/plain",
            CancellationToken.None);

        var metadata = await storage.GetMetadataAsync(key, CancellationToken.None);
        metadata.ContentLength.ShouldBe(bytes.Length);

        await storage.DeleteAsync(key, CancellationToken.None);
        (await storage.ExistsAsync(key, CancellationToken.None)).ShouldBeFalse();
    }
}

Presign contracts must perform real HTTP PUT/GET and assert expiry, type, file name, range requests, 404, authorization revocation, and overwrite behavior. “URL is non-empty” is insufficient.

5. Dual-source failure injection

Current code has no explicit Purge state or checkpoint. These target tests reveal the workflow contracts that still need implementation.

6. Tenant and owner HTTP tests

Build Tenant A/B, User A1/A2, App A, and an Operator:

1. A1’s private file is downloadable only by A1 or an approved Operator;
2. A2 cannot bypass owner policy with ordinary View;
3. App ownership matches ClientId, not UserId;
4. Tenant A public is not downloadable by Tenant B;
5. forged FileId errors do not disclose cross-tenant existence;
6. non-Finalized files cannot download or bind;
7. product explicitly defines whether Delete is tenant-wide or owner-scoped;
8. operate-as records Actor Tenant separately from Effective Tenant.

Create test roles from the real Permission Catalog. Do not inject policy-only permissions that users cannot assign.

7. Concurrency and idempotency

Use a real database and one object to race:

• two logically identical session-create requests;
• two finalize commands, with exactly one state commit and event;
• finalize versus delete, with one legal terminal result;
• download versus delete, defining already issued URL semantics;
• cleanup versus finalize, especially the immediate Pending selection defect;
• two JobHost instances deleting one key;
• repeated provider Delete converging safely.

Assert database, object, event, and audit surfaces, not only HTTP status.

8. Performance and capacity

The hot path is control-plane and storage limits, not API byte proxying:

• session-create P50/P95/P99;
• S3 HEAD and presign latency;
• concurrent uploads and unfinished sessions per tenant;
• bucket object count, bytes, and inventory cost;
• finalize transaction-conflict rate;
• download-grant rate and object GET egress;
• cleanup batch size, provider throttling, and window duration;
• event/outbox backlog.

Load tests must use generated, hash-verifiable, disposable data in a dedicated bucket with its own lifecycle.

9. Metrics

TenantId, UserId, FileId, StorageKey, and FileName must not be low-cardinality metric labels. Put protected identifiers in traces or access-controlled diagnostics with retention limits.

10. SLO and alerts

Calibrate internal objectives to deployment capacity:

• successful session issuance and P95 latency when the provider is healthy;
• finalize success budget after a confirmed PUT;
• Pending count beyond URL TTL plus grace;
• zero Finalized assets whose object is missing;
• Deleted assets whose objects exceed retention;
• cleanup consecutive failures, oldest age, and deletion rate;
• S3 readiness, clock skew, and credential expiry;
• sudden metadata mismatch or cross-tenant denial growth.

Do not page on a single user mistake. Page on systemic mismatch, stopped cleanup, or cross-tenant attack patterns.

11. Troubleshooting flow

Never set Status=Finalized directly in SQL. That bypasses metadata invariants, FinalizedAt, events, and the target security-scanning path.

12. Release drills

For every major upgrade, exercise:

1. real browser CORS upload and finalize;
2. PUT/GET immediately before and after expiry;
3. size, type, and hash mismatch;
4. object inventory after API Save failure;
5. finalize racing cleanup;
6. post-delete object purge;
7. S3 credential rotation and least privilege;
8. API/JobHost provider drift;
9. storage outage, slowness, and throttling;
10. object reconciliation after database restore;
11. tenant-public and cross-tenant attacks;
12. malicious samples in a dedicated security-test environment.

13. GA-blocking gates

• Admission has size ceiling, tenant quota, OwnerType/Visibility registry, and MIME policy;
• presigned upload constrains length, type, and trusted checksum;
• finalize enforces UploadExpiresAt and computes/verifies a server-trusted hash;
• quarantine/scanning states exist; pre-scan content cannot bind or download;
• create/finalize/delete idempotency and concurrency contracts are complete;
• catalog, generic authorization, owner policy, and operate-as agree;
• object deletion, retry, tombstone, and retention converge;
• immediate Pending cleanup and missing Deleted cleanup are fixed;
• API/JobHost provider, bucket, key, and credential rights match;
• S3 and every enabled connector pass one provider contract;
• download does not expose StorageKey and returns actual expiry;
• metrics, alerts, inventory reconciliation, runbook, and ownership are ready;
• dual-ORM, HTTP, security, fault, concurrency, and capacity tests pass;
• documentation separates current, constrained, and target capability.

14. Local verification

# Run Files unit, application, integration, and architecture tests.
dotnet test tests/BitzOrcas.Unit.Tests --filter FullyQualifiedName~FileAsset
dotnet test tests/BitzOrcas.Application.Tests --filter FullyQualifiedName~File
dotnet test tests/BitzOrcas.Integration.Tests \
  --filter "FullyQualifiedName~FileAsset|FullyQualifiedName~LocalFileStorage"
dotnet test tests/BitzOrcas.Architecture.Tests --filter FullyQualifiedName~Files

# Expose current production gaps; expect no key implementation hit before those features land.
rg -n "IFileScanner|UploadQuota|DeleteRequested|Purged|UploadSessionExpired" \
  src tests -g '*.cs'

# Pending should gain an age predicate and Deleted a separate lifecycle path after correction.
rg -n "Status.*PendingUpload|Status.*Deleted|UploadExpiresAt|CreateTime.*cutoff" \
  src/Framework/BitzOrcas.Infrastructure.SqlSugar/DataLifecycle -g '*.cs'

Previous: Storage composition and cleanup · Back to Files