Search Index Events, Version Ordering, and Rebuild

Index correctness comes from repeatedly folding authoritative source facts into documents, not from Lucene alone. Search defines a useful typed change event but has no publisher, document-field provider, or rebuild data source. Its consumer can therefore perform only deletion.

1. Event contract

The type implements IIntegrationEvent without [IntegrationTopic("search.index.changed")]. Source contains event/published/subscribed catalog constants but no construction or typed publisher call.

2. Current consumer state machine

Created/Updated broadcast an index-change notification despite not changing the index. An invalid enum value enters no switch branch, maps to Updated in MapAction, broadcasts, and is acknowledged. Unknown actions should be permanent contract errors sent to quarantine.

3. Supplying a document

Possible contracts:

• full projection in the event: replayable but larger and schema-sensitive;
• identity/version event plus owner callback: owner-controlled but introduces a fallible source lookup;
• owner-published versioned search document: usually the clearest division.

[IntegrationTopic("search.document.changed.v1")]
public sealed record SearchDocumentChangedV1(
    string EventId,
    string TenantId,
    string IndexKey,
    string DocumentId,
    string BusinessType,
    long SourceVersion,
    SearchDocumentAction Action,
    IReadOnlyDictionary<string, string> SearchFields,
    IReadOnlyDictionary<string, string> StoredFields,
    IReadOnlyList<string> AccessTokens,
    DateTimeOffset OccurredAt,
    int SchemaVersion = 1);

// Validate both dictionaries against a per-index schema; arbitrary maps are not a security boundary.
// Deletes need identity/version; create/update must carry a complete rebuildable snapshot.

4. Idempotency and order

EventId and SourceVersion do not currently guard writes. Repeating Delete may be provider-idempotent, but a late Update v8 after Delete v9 would resurrect a document if future code simply Upserts.

Track LastAppliedSourceVersion and a tombstone per tenant/index/document: equal is duplicate, lower is stale, current+1 applies, a higher jump is a repairable gap. An EventId inbox detects duplicate/conflicting payloads. Delete must persist version/tombstone so stale updates cannot resurrect content.

Lucene can store source version, but a relational inbox and filesystem write are not naturally one ACID transaction. Idempotent retry plus reconciliation/rebuild is normally the convergence mechanism.

5. Failure classes

Classify by cause, not action:

• invalid payload/schema/index/action: quarantine/DLQ;
• duplicate/stale: acknowledge with metrics;
• gap: persist repair request;
• transient I/O, lock, source timeout: bounded retry;
• ACL/schema violation: reject and alert;
• notifier failure after a successful document write: retry/repair notification without undoing the document.

Missing Created/Updated infrastructure is an unready system, not a warning that should be acknowledged as success.

6. RebuildIndex current behavior

POST /api/search/index/{indexKey}/rebuild is fully implemented, no longer a placeholder. The RebuildIndex handler injects ISearchEngine, IEnumerable<ISearchIndexRebuildSource>, ISensitiveOperationApprovalPort, and IActivityAuditSink, and performs one atomic full rebuild.

Request-side constraints:

• Requires IdempotencyKey; the command is marked INonTransactionalCommand + IIdempotentRequest.
• ConfirmToken is the fixed literal "REBUILD INDEX" (two words, with a space).
• ExpectedVersion is compared against the version from engine.GetStatisticsAsync before rebuild; a mismatch is rejected, preventing the rebuild from overwriting concurrent changes.
• Authorization is resource search/index, Action Manage; rate limit policy sensitivePolicy, timeout DatabaseMaintenance.

Execution flow:

1. Write the intent audit record.
2. Call engine.RebuildAsync, passing a RebuildDataSource.
3. That source matches an ISearchIndexRebuildSource by IndexKey and streams authoritative batches via a keyset cursor (GetBatchAsync(tenantId, lastCursor, pageSize, ct)); tenant filtering is applied inside the source.
4. On completion, write a completion audit record; an audit failure is treated as fail-closed.

The handler returns a populated RebuildResultDto with before/after document counts and versions. The shadow-rebuild/reconciliation design in section 9 below is a higher-level operational target; this endpoint performs an in-place full rebuild only and creates no shadow directory or atomic alias cutover.

Rebuild data-source contract

GlobalSearchContracts.cs defines the server-trusted contracts rebuild depends on:

• ISearchIndexRebuildSource: an authoritative batch source provided by a business owner. Members are string IndexKey and Task<IReadOnlyList<SearchIndexRebuildDocument>> GetBatchAsync(string tenantId, string? lastCursor, int pageSize, CancellationToken). IndexKey is decided by server registration; clients cannot submit arbitrary index keys.
• IGlobalSearchSourceCatalog: member IReadOnlyList<GlobalSearchSourceDefinition> GetSources(), enumerating registered global search-source definitions.

Current concrete implementations: TicketSearchIndexRebuildSource and WorkflowTaskSearchIndexRebuildSource, for index keys tickets and workflow-tasks. An unregistered IndexKey fails the match stage and is not treated as a rebuildable scenario.

7. Administration overview endpoint

GET /api/search/admin/overview returns a cross-index operational snapshot. Authorization is resource search/index, Action View; rate limit userPolicy, timeout ShortRead.

It returns SearchAdministrationOverviewDto:

Each SearchAdministrationItemDto carries IndexKey, DisplayName, EntityType, Status ("ready", "read-only", or "unavailable"), DocumentCount, Version, CanRebuild (bool), and ErrorCode (string?).

The data is aggregated from IGlobalSearchSourceCatalog registrations, ISearchIndexRebuildSource registrations, and the live result of engine.GetStatisticsAsync per index. An exception from any one source degrades only that item (marked as degraded) without affecting the others, so IsPartial being true still returns the partial set already collected.

8. Data-source contract

Every IndexKey source needs an authoritative owner/schema version, tenant partitions, keyset cursor, consistent high-watermark, complete document/ACL projection, tombstone semantics, cancellation/backpressure, and count/hash reconciliation. Search must not reflect over business tables; owners expose narrow snapshot streams through Contracts.

9. Shadow rebuild

Validate catalog and disk, create a unique shadow path, capture W, stream by tenant, buffer live delta, catch up, force commit, reconcile, atomically switch, observe, retain the old version for rollback, then clean it.

The Lucene package README describes IndexNew/Bak/Temp replacement. The platform does not call Rebuild or orchestrate these states; package capability is not module-level operational readiness.

10. Rebuild while querying

Define old-reader continuity, the instant new queries see the version, in-flight behavior, Windows/Linux rename semantics, crash recovery for Temp/Bak, doubled disk budget, and mutual exclusion for concurrent rebuilds.

Useful states are Active → RebuildQueued → Rebuilding → CatchingUp → Verifying → Switching → Active, with Failed retaining the previous Active. No such platform state machine or lock exists today.

11. Replica topology

Default BasePath is a process-relative search-index. With multiple API replicas, each local volume differs, a CAP consumer group normally sends an event to one instance, and invalidation does not copy a Lucene document. Queries can therefore vary by destination replica.

Choose a single-writer Search service, complete fan-out to every replica, a centralized external provider, or another validated topology. Multiple Lucene writers on ordinary shared NFS are not a safe default without provider guarantees and stress tests.

12. Reconciliation and erasure

Compare source count, ID/content hashes, tombstones, ACL tokens, and orphans by tenant/index/version. Repair a document or trigger shadow rebuild. Privacy erasure and access revocation must reach every active/shadow/backup version; rollback and old snapshots must not restore erased content.

13. Required event/rebuild tests

1. Real owner-published bytes bind to the typed contract.
2. Created/Updated produce complete schema documents.
3. Deleted retries and writes tombstone semantics.
4. A,A and A,B,A replay.
5. Update v8, Delete v9, late Update v8.
6. Version-gap repair.
7. Provider and notifier fail independently.
8. Rebuild + live delta + atomic cutover.
9. Crash at every commit/rename phase.
10. Identical results across tenants and API replicas.
11. ACL revocation and erasure across old versions.
12. Source/index reconciliation locates drift.

14. Review commands

# A consumer and catalog exist, but no producer or IntegrationTopic is present.
rg -n "SearchIndexChangedIntegrationEvent|search\.index\.changed|IntegrationTopic|PublishAsync" \
  src/Platform/Search src/Platform -g '*.cs'

# Created/Updated are no-ops; SourceVersion is forwarded rather than applied.
rg -n "SearchIndexAction|UpsertAsync|DeleteAsync|SourceVersion|NotifyIndexChangedAsync" \
  src/Platform/Search -g '*.cs'

# Rebuild is implemented: expect ISearchIndexRebuildSource, RebuildDataSource, audit, and ExpectedVersion comparison.
rg -n "ISearchIndexRebuildSource|RebuildDataSource|ExpectedVersion|RebuildAsync|SensitiveOperationApproval" \
  src/Platform/Search tests -g '*.cs'

Back to Search · Provider and catalog · Testing and GA