BitzOrcas runs scheduled and batch work in the separate BitzOrcas.JobHost process. The API Host remains trim-safe and does not reference Quartz. JobHost owns scheduling; modules own the actual business executors.

Module declaration and executor → JobHost descriptor → Quartz trigger
                                      ↓
                              Audit envelope and OTel
                                      ↓
                              Owner-local business port

Critical path diagram

Follow the main path to identify responsibility handoffs, then use the prose to inspect failure branches and evidence.

When to use a job

Good candidates include periodic cleanup, aggregation, backup verification, key rotation, timeout scanning, and accepted asynchronous work such as export execution. Do not use a job for an action that must commit atomically with the current transaction or a short command whose caller is waiting.

JobHost is not limited to audit retention. It composes work owned by Audit, Workflow, Identity, Backup, Export, DataLifecycle, Website, and Payment, including export execution, workflow timers, orphaned-file cleanup, CAP cleanup, auto-renewal, and backup retention.

Three parts of a job

BackgroundJobDeclaration: stable name, cron or interval, module, enabled state, criticality, configuration source, and description.
IJobExecutor: a Quartz-neutral business entry point whose expected failures return Result.Failure.
JobHost adapter: maps a Quartz firing to the executor and applies the common audit envelope.

An enabled declaration must provide exactly one cron expression or positive interval. A disabled job may omit its schedule, while its Job type remains registered for controlled manual execution.

Adding a job

Define a stable identity and declaration in the owning module.
Implement IJobExecutor<TIdentity> using module ports and the supplied CancellationToken.
Bind declaration, executor, and Quartz adapter in that module’s JobHost registration extension.
Test success, expected failure, exception, cancellation, duplicate execution, and concurrent execution.
Confirm that the Operations schedule report and JobHost health checks include the new job.

Do not grow Program.cs with manual AddJob<T>() calls. AddJobHostBackgroundJobs gathers owner-local extensions and validates that Catalog declarations, handlers, and executors are complete and unique at startup.

Idempotency and concurrency

Schedulers commonly create at-least-once execution conditions: a process may exit after the business write but before acknowledging completion, and operators may rerun work. An executor therefore needs a durable way to recognize an already completed batch or business key.

Typical controls include unique constraints, conditional state transitions, processing cursors, and idempotency records. Do not rely only on the assumption that cron firings never overlap. Multi-instance JobHost uses persistent Quartz clustering, but business logic must still tolerate duplicate triggers.

Tenant boundary

For multi-tenant batches, obtain an explicit tenant list and establish trusted context for each tenant. Queries, cursors, locks, and idempotency keys all need tenant dimensions. A worker has no browser user and must not fabricate a normal user context to bypass authorization.

Failure, audit, and cancellation

QuartzJobExecutionAuditor creates an Activity and records job name, Quartz FireInstanceId or a generated correlation identity, duration, and final status. Result.Failure becomes a host failure so Quartz and audit see the real outcome; system exceptions and cancellation continue to propagate.

An audit-sink failure produces a warning and does not turn business success into failure. Conversely, a business exception must not be swallowed. Long loops should observe cancellation regularly so deployment shutdown can converge.

Declaration, configuration, and runtime report

Stable job identity and default schedule belong to the module contract; environment enablement and Cron/interval overrides belong to deployment. A name used by alerts, operations APIs, and audit cannot change with a class refactor.

// A declaration contains stable scheduling metadata, never business execution logic.
public static readonly BackgroundJobDeclaration Cleanup = new(
    // Configuration, audit, and operations APIs share this stable name.
    Name: "files.orphan-cleanup",
    Module: "Files",
    CronExpression: "0 0/15 * * * ?",
    Interval: null,
    Enabled: true,
    Critical: false,
    ConfigurationSource: "Files:Jobs:OrphanCleanup");

The Operations report should expose declaration, actual trigger, next fire, recent result, and mismatch reason. An enabled declaration without a trigger, duplicate name, missing executor, or simultaneous Cron/interval is a startup or GA blocker.

Batch cursor and transaction boundary

Process bounded batches and commit a cursor instead of holding an hour-long database transaction. Advancing before the business write loses data; writing before advancing may replay, so the operation must be idempotent.

// Advance only after the business batch commits; business keys absorb replay.
while (!cancellationToken.IsCancellationRequested)
{
    // Bounded reads constrain transaction time, memory, and shutdown latency.
    var batch = await source.ReadAsync(tenantId, cursor, batchSize: 200, cancellationToken);
    if (batch.Count == 0) break;

    await processor.ApplyIdempotentlyAsync(batch, cancellationToken);
    cursor = await cursors.CommitAsync(tenantId, batch[^1].Sequence, cancellationToken);
}

Do not pretend an external call and database transaction are atomic. Use an Outbox after database changes. For a provider call, persist recoverable state and a provider idempotency key.

Retry, misfire, and overlap

Scheduler retry, the next Cron scan, and manual replay may target the same fact but need different backoff and alerts. Select misfire behavior by domain: settlement may need catch-up, while periodic refresh often needs only the newest run.

Quartz locking constrains scheduler instances, not every manual endpoint or old process. Use leases, unique batches, or conditional updates. A lease needs owner, expiry, and fencing token rather than a permanent Boolean lock.

Tests and fault injection

terminate after the business write but before cursor commit; restart causes no duplicate effect;
fire the same trigger concurrently and commit one batch effect;
isolate one tenant failure while other tenants continue;
cancellation stops new batches and safely completes the current unit;
audit/OTLP failure preserves business result but raises operations evidence;
validate time zone, daylight-saving, misfire, and next-fire behavior.

Release gate

Production database configuration exists and the JobHost runtime guard passes.
Critical jobs are enabled, have a future firing, and scheduler health is green.
Rerunning work cannot duplicate billing, messages, or destructive actions.
Multi-tenant scans have no cross-tenant reads or shared cursors.
Failure records, Activities, alerts, and controlled manual retry are available.

See Jobs building block for types and composition details.