Search Lucene Provider, Catalog, and Segmentation

Search Infrastructure composes an external engine as ISearchEngine, implements catalog/segmentation ports with owner-local tables, and supplies platform stop words. A constructible provider, an existing catalog entry, and populated index data are three separate facts.

1. Provider manifest

Blank Search:Provider selects Lucene. Unknown and unimplemented values fail instead of silently falling back.

{
  "Search": {
    "Provider": "Lucene",
    "LuceneBasePath": "/var/lib/bitzorcas/search"
  }
}

Without BasePath, the relative search-index directory is used. Production needs an explicit absolute persistent volume, permissions, capacity/inodes, backup/rebuild policy, and single-writer ownership.

2. External package boundary

Directory.Packages.props pins 1.0.0-alpha.8. The core package defines the engine, input pipeline, data-source, word, and catalog abstractions. The Lucene package implements analyzers, query/write/delete, delayed commit, and directory replacement capabilities.

Package README features are not automatically BitzOrcas GA guarantees. Platform composition, schema, tenancy, operations, and upgrade compatibility need independent consumer tests before an alpha upgrade.

3. Composition and fail-closed defaults

// Generated defaults provide unavailable ports when the host lacks required capabilities.
services.AddBitzOrcasGeneratedPersistenceDefaults();

// The selected database adapter provides IEntitySet for the two owner-local records.
services.AddBitzOrcasGeneratedPersistenceAdapters(persistenceProvider);

// Provider resolution then registers Lucene, stores, stop words, and the CAP consumer.
services.AddBitzOrcasSearchPlatform(configuration);

ISearchEngine requires Search capability; ISegmentWordStore requires Database and Search. Architecture/smoke tests guard defaults and production override.

4. SearchIndexCatalog is metadata, not the index

No catalog seed or create use case exists. A new database may have the table and return an empty catalog while the Lucene engine still accepts an arbitrary IndexKey, allowing physical/catalog drift.

5. Catalog Store semantics

GetAll has no ordering; GetAsync matches IndexKey exactly. UpdateAsync changes an existing row only and silently no-ops when missing.

// The persisted string may come from an older release, manual write, or newer provider state.
var status = Enum.TryParse<IndexStatus>(entity.Status, out var parsed)
    ? parsed
    : IndexStatus.Active;

// This preserves list availability but can misreport corrupt/new status as usable.
return new IndexCatalogEntry(
    entity.IndexKey,
    entity.DisplayName,
    entity.DirectoryName,
    entity.EntityType,
    status,
    entity.DocumentCount,
    ToUtcOffset(entity.LastRebuiltAt),
    entity.Version);

Management should expose Unknown/Unavailable and alert. LastRebuiltAt is stored as DateTime? and restored with zero offset; test Kind and precision in both ORMs.

6. Global definition versus tenant state

The catalog has no TenantId, suggesting a global scenario definition with tenant-partitioned documents. Yet DocumentCount, Version, and LastRebuiltAt do not say whether they are global, per tenant, or aggregated. Statistics explicitly is tenant-scoped.

Separate global SearchIndexDefinition, tenant SearchIndexPartitionState, and per-run SearchIndexRebuildRun. One global row cannot accurately express all three.

7. SysSegmentConfig scope and writes

The table is global. SegmentType represents LegalSuffix, OrgWeak, or GeoWeak; uniqueness is (SegmentType,Content) and Content length is 200.

Add performs exact check-then-insert. Concurrent identical first writes can race on the unique key, and the handler does not map that conflict to idempotent success. Delete and list use exact content; list is unordered. Input is trimmed but not Unicode-normalized or case-folded, while the in-memory bucket is ordinal-ignore-case. Database collation can differ.

8. Scoped cache reality

The store is registered Scoped and its cache is an instance field. The same scope serializes initial load with SemaphoreSlim; Add/Delete invalidates only that instance. A new request/replica owns another cache, and RefreshCache refreshes the current instance only.

This is not a distributed hot cache. Verify the actual DI graph and benchmark table reads before changing lifetime.

9. Stop-word provider

EmbeddedStopWordProvider returns a fixed Chinese/English punctuation set and performs no I/O. It is global. Stop words differ from semantic weak words: changing legal suffixes or geography terms can alter conflict results and therefore needs versioning, approval, impact analysis, rebuild, and audit.

The public type is IReadOnlySet, but the backing object is a static mutable HashSet. A frozen/immutable collection would make the guarantee stronger.

10. Dual-ORM behavior

Both records use generated metadata and stores depend only on IEntitySet<T>. Architecture tests prove no concrete ORM dependency. In-memory unit tests do not prove unique-conflict mapping, collation, DateTime Kind/precision, invalid enum values, concurrent add/delete/load, or stable ordering.

Run the same behavioral contract against real SqlSugar and EF Core providers.

11. Operations checks

# Production should use an explicit absolute path rather than the process working directory.
test -n "$Search__LuceneBasePath"
test "${Search__LuceneBasePath#/}" != "$Search__LuceneBasePath"

# Validate writable storage, capacity, and inodes without printing indexed content.
test -d "$Search__LuceneBasePath" && test -w "$Search__LuceneBasePath"
df -h "$Search__LuceneBasePath"
df -i "$Search__LuceneBasePath"

Mount a persistent volume in containers and test old/new package read compatibility before rolling updates. Do not delete a directory as “rebuild” until the data-source path and recovery duration have been proven.

12. Review commands

# Review manifest, configuration keys, and alpha package versions together.
rg -n "SearchProviderKind|IsImplemented|Search:Provider|LuceneBasePath|Bitzsoft.Integrations.Search" \
  src/Platform/Search Directory.Packages.props -g '*.cs' -g '*.props'

# Catalog has no current seed/create path.
rg -n "SearchIndexCatalogRecord|ISearchIndexCatalogStore|UpdateAsync\(" \
  src tests -g '*.cs'

# Segmentation is a global record with a scoped instance cache.
rg -n "SearchSegmentConfigRecord|TryAddScoped<ISegmentWordStore|_loaded|RefreshCacheAsync" \
  src/Platform/Search tests -g '*.cs'

Back to Search · Events and rebuild · Testing and GA