Search Testing, Observability, and GA Gates

The most dangerous Search failure is often a successful response that is incomplete, unauthorized, stale, or incorrectly says “no conflict.” GA acceptance must prove result trust, not only DI resolution and a mocked handler call.

1. Current evidence boundary

Existing tests prove:

• ListSegmentWords delegates to its store;
• an in-memory catalog row can be read and updated;
• a serial segmentation add/list/delete/cache path works;
• owner-local records and generated metadata exist;
• Search Infrastructure is ORM-neutral;
• host references, API composition, provider manifest, and unavailable defaults are present.

They do not prove real Lucene tenant isolation/ranking, endpoint authorization, event binding/replay/indexing, StoredFields/Total security, safe conflict decisions during lag/failure, rebuild, replica consistency, corruption recovery, or privacy erasure.

2. Acceptance layers

Treat the external package as a consumer contract. Its own tests cannot prove platform configuration, schema, tenancy, or upgrade behavior.

3. Real Lucene suite

Use an isolated temporary directory per test and clean it reliably. Cover:

1. same DocumentId across tenant A/B;
2. Upsert, update replacement, idempotent Delete;
3. Name/Subject/Content, Chinese, pinyin, and NGram;
4. legal suffix/organization/geography weak words and stop words;
5. Skip/Take, Total, deterministic equal-score ordering;
6. StoredFields null/type/length/Unicode;
7. visibility before and after delayed commit;
8. concurrent readers/writers and file locks;
9. corrupt, full, and read-only directories;
10. package-version index compatibility;
11. cancellation and timeout;
12. continuous queries during cutover.

[Fact]
public async Task SameDocumentId_InDifferentTenants_MustNeverCrossRead()
{
    await using var fixture = await LuceneFixture.CreateIsolatedAsync();

    // Deliberately reuse the business id while changing tenant and sensitive text.
    await fixture.Engine.UpsertAsync(Document("tenant-a", "case-42", "Alpha secret"));
    await fixture.Engine.UpsertAsync(Document("tenant-b", "case-42", "Beta secret"));
    await fixture.ForceCommitAsync();

    // Neither hits nor StoredFields from tenant B may appear for tenant A.
    var result = await fixture.SearchAsync("tenant-a", "Beta");
    result.Hits.ShouldBeEmpty();
    result.TotalHits.ShouldBe(0);
}

4. Wire and projection sequences

Start from real owner outbox bytes, not a manually constructed consumer object. Pin topic, envelope, schema, enum encoding, SourceVersion, and missing-field failure.

Created/Updated no-op and missing publisher make these tests fail first; that is the correct evidence.

5. Authorization and leakage suite

• exact .check/.search and .manage/.create/.update/.delete outcomes;
• Feature-disabled denial;
• unified search cannot bypass conflict permission;
• TenantId is validated at ingestion and query;
• owner ACL/DataScope revocation removes visibility;
• StoredFields allowlist/masking;
• hidden hits do not leak through Total, hasMore, or timing;
• global catalog/word administration requires platform scope;
• logs, traces, and Problem Details omit terms, parties, and physical paths.

Use the production authorization pipeline and real per-index schema.

6. Conflict acceptance

Test exact/former/foreign names and credit code; same-name false positives; abbreviation/pinyin/weak/geography terms; party roles; exact ExcludeCaseId; duplicate-party handling; stale-watermark behavior; provider failure; review/override states; rule/analyzer/dictionary replay; and privacy retention/export/erasure.

Maintain a legal-reviewed golden corpus with versioned expected matches and tolerances. Algorithm upgrades need a differential report, not only a pass percentage.

7. Dual-ORM store suite

Run the same SqlSugar and EF Core cases for catalog read/update, unknown status, UTC precision, missing-row update, segment uniqueness/collation/Unicode, concurrent Add, idempotent Delete, stable list order, invalid type, large-dictionary load, and cancellation.

The current in-memory set does not exercise relational unique constraints, collation, or time conversion.

8. Observability

Logs should carry allowed tenant/index/document identifiers or hashes, EventId, SourceVersion, index version, result, duration, and trace id. Never full terms, parties, or StoredFields.

9. SLO and freshness

Beyond query latency, define a freshness SLO, for example “an owner change is queryable within 60 seconds.” Expose index version, source watermark/last occurred-at, lag, rebuild/degraded status, authorization mode, and—on conflict checks—rule/dictionary version.

Ordinary search may return an explicit degraded result when stale. A compliance decision should return indeterminate/fail-closed, never false conflict, after its freshness budget expires.

10. Incident response

Search always returns zero

Separate valid zero from missing ingestion. Inspect producer/outbox, consumer applied count, Lucene count, and watermark. In current source no producer exists and Create/Update no-op; restarting API cannot backfill.

API replicas disagree

Capture serving instance and index version. Inspect each BasePath/volume. If one consumer instance receives each event, local indexes necessarily diverge; stop routing to incomplete replicas and rebuild according to the chosen topology.

Lucene directory is corrupt or locked

Stop writers and preserve a read-only diagnostic copy. Rebuild from a validated source into shadow, reconcile, and cut over. Without a source, treat it as data-loss impact rather than claiming easy rebuild.

Suspected missed conflict

Freeze related intake decisions, capture index/rule/dictionary version and watermark, verify authoritative sources, repair ingestion/rebuild, rerun affected checks, and retain old/new conclusions and human actions.

11. GA gates

• owner production contract and byte tests pass;
• every action has version/inbox/gap behavior;
• data source, shadow rebuild, reconciliation, and rollback are rehearsed;
• permissions, Feature, IndexKey, and hit-level authorization align;
• StoredFields schema and sensitive-field policy are enforced;
• conflict checking has records, review, override, audit, and freshness fail-closed;
• real Lucene, dual-ORM, nine-endpoint, and replica tests pass;
• absolute storage path, capacity, writer ownership, and topology are explicit;
• metrics, alerts, SLO, runbooks, and on-call ownership exist;
• alpha package upgrades have compatibility and rollback evidence;
• erasure/revocation reaches active, shadow, and backups;
• every “supported” statement points to source, test, or operational evidence.

Current Search is not GA for business indexing or legal conflict decisions. A registered Lucene provider is foundational capability, not an end-to-end guarantee.

12. Review commands

# Current Search behavior tests are sparse; target suites cover provider, consumer, endpoints, security, and rebuild.
rg -n "Search|Lucene|Conflict" tests -g '*.cs'

# Expose no-op actions, suppressed failures, fixed rebuild failure, and missing publisher.
rg -n "deferred|suppress|RebuildPendingDataSource|SearchIndexChangedIntegrationEvent" \
  src/Platform/Search src -g '*.cs'

# Review production path and host topology.
rg -n "AddBitzOrcasSearchPlatform|LuceneBasePath|search-index|SearchIndexEventHandler" \
  src/Hosts src/Platform/Search -g '*.cs'

Back to Search · Unified search and conflict · Events and rebuild · Provider and catalog