Operations Background-Job Runtime Control

The background-job control plane spans three facts: generated IBackgroundJobCatalog declarations, persisted SysBackgroundJobDefinition rows, and the observed JobHost/API-fallback scheduler state. Operations can edit definitions and execute jobs, but query and scheduler convergence are not one strongly consistent system.

1. Three facts

Code owns JobName, Source, Description, ModuleName, and IsCritical. Persistence owns runtime Cron/Interval/Enabled. Scheduler hosts consume persisted state.

2. Current query source

OperationsService.GetJobScheduleAsync iterates _jobCatalog.Declarations, sorts by JobName, and projects them. It does not read IBackgroundJobDefinitionStore.

After a successful PUT or stop, GET /api/operations/jobs may still show code defaults. A test name and handler comment claim persisted values, but the handler substitutes IOperationsService; it does not prove the concrete service reads the store.

This is source/test-intent drift, not a guarantee.

3. Schedule representation

The DTO field remains CronExpression:

• Cron declarations return their expression;
• interval declarations return interval:{seconds}s;
• unscheduled declarations return null.

Clients must recognize the prefix instead of passing every non-empty value to a Cron parser.

4. Persistent aggregate invariants

BackgroundJobDefinition owns compile-time ORM metadata for SysBackgroundJobDefinition. JobName is unique and the row is soft-deletable.

Runtime rules are:

• Cron and Interval cannot both be set;
• Interval must be positive;
• Enabled requires one schedule;
• Disabled may have no schedule;
• Cron is trimmed and limited to 160 characters but not syntax-validated.

5. Update a schedule

var command = new UpdateBackgroundJobDefinition.Command(
    JobName: "export-execution",
    CronExpression: null,
    IntervalSeconds: 45,
    Enabled: true);

// ① Store finds a non-deleted definition by stable JobName.
// ② The aggregate validates and replaces all three runtime fields.
// ③ SaveAsync joins the current unit of work and returns interval:45s.
Result<JobScheduleEntry> result =
    await mediator.Send(command, cancellationToken);

Update cannot create arbitrary jobs. A definition must first come from the code catalog and seed process.

6. Start and stop preserve schedule

SetBackgroundJobEnabled loads the definition, retains Cron/Interval, and changes only Enabled.

Starting an unscheduled disabled definition fails. PUT a valid schedule before enabling it.

7. HTTP examples

POST /api/operations/jobs/workflow-timer/stop
Authorization: Bearer <token-with-operations.jobs.manage>

PUT /api/operations/jobs/export-execution
Content-Type: application/json
Authorization: Bearer <token-with-operations.jobs.manage>

{
  "cronExpression": null,
  "intervalSeconds": 60,
  "enabled": true
}

The handwritten endpoint group applies authentication, userPolicy rate limiting, and the exact management permission.

8. Manual execution path

ExecuteBackgroundJob.Command implements INonTransactionalCommand, avoiding a generic database transaction around long-running external work.

The handler confirms the definition exists, sends the stable JobName to IJobExecutorDispatcher, and wraps it with IBackgroundJobExecutionAuditor.

9. Dispatcher closure at startup

JobExecutorDispatcher rejects:

• duplicate bindings for one JobName;
• a catalog declaration with no binding;
• a binding absent from the catalog.

It throws during construction, exposing composition errors early. Execution uses the validated dictionary instead of scanning DI.

10. Execution audit semantics

The envelope records timing, success/failure, error code, and correlation. Business failure is returned unchanged. Exceptions and cancellation are audited as failure and then rethrown.

Audit sink failure produces a warning and does not alter job outcome. That availability-first behavior needs a separate audit-health gate in regulated deployments.

11. Schedule-edit audit gap

Update/start/stop have no explicit before/after activity audit. They depend on the generic command pipeline to capture enough evidence, and this handbook cannot promise old value, new value, actor, and reason are persisted together.

There is also no approval, reason, version, or concurrency token. Concurrent administrators are last-writer-wins.

12. Multi-instance propagation

A successful Store update means desired state is persisted. Whether and when Quartz or API fallback changes depends on each host’s reload/poll path.

The response contains no scheduler acknowledgement, configuration version, or effective time. A commercial console needs desired-versus-observed state.

13. Manual execution while disabled

The handler checks existence but not Enabled. Stopping automatic scheduling does not prohibit an administrator from executing the job manually.

Treat Enabled as an automatic-schedule switch, not a global kill switch. Add a separate suspension state if product semantics require one.

14. Troubleshooting

NotFound: inspect catalog, seed execution, soft deletion, and the target database.

ScheduleConflict: both Cron and Interval were supplied.

ScheduleRequired: enabling a definition with no schedule.

Execution failure: inspect BackgroundJob audit before the binding’s domain error; missing audit also requires checking sink health.

GET differs from PUT: the concrete GET currently reads the code catalog.

15. Test focus

Existing tests cover schedule replacement, invalid-update no-write, start/stop preservation, Manage authorization, successful/business-failed/exception/canceled execution audit, HTTP 403, and management routes.

Missing coverage includes concrete GET-after-PUT, Cron syntax, concurrent updates, scheduler convergence, multi-instance reload, audit-health alerting, disabled manual execution policy, timeout, and cancellation.

16. GA order

1. GET persisted desired state alongside catalog defaults.
2. Add observed scheduler state and instance heartbeat.
3. Add definition version/ETag to reject lost updates.
4. Parse Cron and preview next fire times.
5. Audit before/after values, reason, and optional approval.
6. Publish reload and wait for instance acknowledgements.
7. Add idempotency, timeout, cancellation, and concurrency policy to manual runs.
8. Run one contract suite against JobHost and API fallback.

17. Test commands

# Application-level schedule and execution behavior.
dotnet test tests/BitzOrcas.Application.Tests \
  --filter FullyQualifiedName~BackgroundJobManagementTests

# Handwritten job-management HTTP routes and permissions.
dotnet test tests/BitzOrcas.Integration.Tests \
  --filter FullyQualifiedName~BackgroundJobManagementApiTests

# Catalog, binding, and runtime-surface closure.
dotnet test tests/BitzOrcas.Architecture.Tests \
  --filter 'FullyQualifiedName~BackgroundJobIntakeTests|FullyQualifiedName~OperationsRuntimeSurfaceTests'

Operations overview · Security and GA