Billing Renewal, Grace Sweep, Reconciliation, Testing, and GA

JobHost registers auto-renewal, grace-sweep, and reconciliation. All three Quartz jobs use [DisallowConcurrentExecution] and the common QuartzJobExecutionAuditor, but scheduling and audit wrappers do not make collection, entitlement extension, expiry handling, or difference remediation complete.

1. Scheduling

All three default to disabled. JobHost registers the Billing repository and payment gateway, with fail-closed ports when no provider is configured.

[DisallowConcurrentExecution] protects a JobDetail as understood by Quartz. Cross-node exclusion depends on the actual Quartz store/cluster topology. The Billing executor has no owner-local distributed lease or fencing token.

2. Actual renewal flow

The repository selects:

Status in (Active, Grace)
AND EndedAt IS NOT NULL
AND EndedAt <= now + ExpiryWindowDays

For each subscription, the executor loads the Plan, creates and issues an auto-renewal:{subscriptionId} invoice for the current yyyy-MM, saves it, marks a free invoice Paid or creates a provider order, then advances EndedAt by one month and saves the subscription. Ordinary Result failures enter Grace, processing continues, and the first failure is returned at the end. Unexpected exceptions and notification failures propagate and stop that execution.

3. Renewal GA blockers

Order creation grants another month

A provider accepting an order does not mean the customer paid. The executor still calls Renew. Payment callbacks later mark Invoice Paid but do not grant the renewal. A prepaid model must keep Pending/Grace until verified payment; a postpaid model needs an explicit credit decision, due date, collections, and suspension policy.

The payment handle is discarded

PayUrl, QrCode, PrepayId, and provider transaction context are not stored or delivered to the tenant. Interactive Web/Code orders may be impossible for the customer to complete.

Publicly started subscriptions are not scanned

StartSubscription sets EndedAt to null, while the scan requires non-null. No public use case initializes the first end date.

The batch reports its first failure but has no item ledger

Ordinary Result failures no longer look like a successful batch. The executor continues with later subscriptions and returns the first failure; an EnterGrace persistence failure also propagates through that result. Unexpected exceptions and notification failures are thrown, so later items are not processed.

This is safer than silent success, but there is still no claim, checkpoint, or durable per-item execution record. A retry rescans the source and cannot distinguish not-started, ordered, renewed, and notification-failed intermediate states.

Renewal idempotency has no durable target-period attempt

The invoice purpose is fixed at auto-renewal:{subscriptionId}. The invoice key also includes TenantId and Period, so ordinary months remain distinct, but the executor does not persist a RenewalAttempt keyed by target period and expected subscription version. If invoice or order creation succeeds and subscription persistence fails, cross-process recovery is not complete.

Free renewal emits no paid event

Free plans call MarkPaid and Save directly, bypassing PaymentGatewayAdapter and InvoicePaidIntegrationEvent. Downstream event consumers see different free and paid lifecycles.

4. Safer orchestration

Persist a RenewalAttempt uniquely keyed by SubscriptionId and target period. Store expected EndedAt/Version so retries cannot extend multiple months. Invoice idempotency alone does not make subscription.Renew idempotent.

5. Actual grace-sweep semantics

grace-sweep selects:

Status == Grace
AND EndedAt IS NOT NULL
AND EndedAt <= now - GraceWindowDays

For each item it calls Expire(now), saves the subscription, and publishes platform-billing.subscription.expired. Ordinary Result failures preserve the first error and allow the scan to continue. Repository-query failures, unexpected exceptions, and notification failures fail the job.

Unit tests cover disabled execution, expiry plus notification, persistence failure, and the within-window boundary. The path still has no multi-instance claim or lease, batch limit, pagination, checkpoint, or durable failed-item queue. Product semantics also need attention: ResumeSubscription can move Grace directly back to Active without verified payment, so a scheduled expiry sweep does not by itself prove successful collection.

6. Reconciliation only reports

The job scans all Issued and Overdue invoices and queries each provider by IdempotencyKey:

• query failure: warning and +1 difference;
• not found: warning and +1;
• provider Success while platform is unpaid: warning and +1;
• Success with a different amount: another warning and +1;
• final log contains DifferenceCount; Result is still Success.

It does not mark Paid, persist a report, notify, create a ticket, suspend, page, throttle, or checkpoint. A successful-but-wrong-amount invoice counts twice. The adapter divides by 100 and does not compare currency.

var result = await executor.ExecuteAsync(cancellationToken);
if (result.IsFailure)
    return result.Error; // Usually an entire repository scan failure.

// Success means the scan finished. It does not mean zero differences.
// DifferenceCount is present only in logs.
return Result.Success();

7. Operational reconciliation model

A ReconciliationRun should persist provider/account, window, timestamps, scanned/matched/difference/failure counts, checkpoint, and software version. Each difference needs platform/provider status and money, currency, transaction identifiers, classification, evidence hash, owner, SLA, remediation state, and audit.

Automatic posting is safe only for strong evidence: provider success, matching merchant/order/amount/currency, and an Issued platform invoice. Amount mismatch, unknown payment, or payment after Void requires manual review.

8. Observability

Track renewal scanned/succeeded/grace/exception, grace scanned/expired/failure/notification, provider order latency/errors, renewal-before-payment count, invoice age by state, verified/rejected/duplicate callbacks, reconciliation scanned/differences/unresolved, InvoicePaid Outbox lag, and Job last-success/duration/next-fire. Do not use raw tenant, invoice, or trade numbers as unbounded metric labels.

9. Failure runbooks

Renewal orders fail in bulk

Disable auto-renewal, inspect provider credentials/certificates/network/limits, query already-created orders, and avoid blind reordering. The current absence of RenewalAttempt requires exporting invoice, subscription, and log evidence before remediation.

Callback success rate drops

Split signature, status, amount, not-found, persistence, and Outbox failures. HTTP 2xx is not enough because hard failures also return 2xx. Query the provider and assess the time window in which genuine payments may have been acknowledged but not posted.

Provider is Paid while platform is not

Pause automatic void/collections. Verify merchant, OutTradeNo, amount, currency, provider transaction ID, and callback evidence. Compensation must use the same conditional state update and Outbox transaction as the callback and record operator/reason.

A rerun extended multiple months

Pause renewal and reconcile invoice purpose/period against EndedAt. Do not silently rewrite time. Introduce a target-period attempt and expected subscription version.

10. Current test evidence

Present: aggregate state machines, Plan update, sequential monthly-invoice idempotency, sequential usage deduplication/quota rejection, entitlement cache-key shape, invoice-key composition, read-handler delegation, persistence architecture, and seed tests. AutoRenewal covers invoice-save failure, free Paid-save failure, subscription-save failure without a false notification, and notification exceptions. GraceSweep covers disabled execution, expiry plus notification, save failure, and window boundaries.

Absent: payment adapters/endpoints/provider fixtures, the Reconciliation executor, Quartz composition, invoice+CAP fault injection, real dual-ORM repository behavior, all nineteen HTTP routes and permissions, concurrent invoices/usage/callbacks/renewal, provider sandbox tests, key rotation, recovery, batching, throttling, and performance baselines.

11. Recommended test pyramid

1. Domain invariants and state-transition tables.
2. Every command/query Result, tenant, permission, and transaction effect.
3. Three-provider mapping and signed callback byte fixtures.
4. Dual-ORM uniqueness, concurrency Version, append-only facts, and tenant filters.
5. InvoiceIssued/Paid byte contracts, Outbox retry, and duplicate consumption.
6. Nineteen routes, Problem Details, anonymous callback, rate limit, and timeout.
7. All three job windows, partial failure, idempotent rerun, and multi-instance exclusion.
8. Provider sandbox order→callback→Paid→renewal.
9. Provider/DB/CAP outages, callback backlog, and reconciliation recovery drills.
10. Large-tenant usage, invoice query, renewal batch, and provider-rate-limit baselines.

12. Commercial GA gate

• approved pricing, period, money, rounding, tax/discount, and refund boundaries;
• complete first-period, end-date, trial, change, cancellation, and history semantics;
• explicit prepaid/postpaid policy with no accidental unpaid entitlement;
• PaymentAttempt, callback inbox, provider transaction identity, and concurrent idempotency;
• provider/merchant/order/amount/currency validation and key rotation;
• fault-injected atomicity/compensation across invoice, payment fact, Outbox, and renewal;
• persisted reconciliation, alerts, owner, SLA, compensation, and manual review;
• atomic quotas, windows, archive strategy, and production-scale performance;
• active permission, Feature, DataScope, audit, and privacy controls;
• traceable dual-ORM, provider, API, message, Job, recovery, and load-test evidence;
• runbooks, dashboards, on-call ownership, and sandbox-to-production checklist.

The domain skeleton and callback transaction are useful foundations. The current renewal, reconciliation, and concurrency evidence are insufficient for commercial billing GA.

13. Review commands

# Locate the exact renewal-order and report-only reconciliation behavior.
rg -n "FindExpiringSubscriptions|FindGraceSubscriptions|CreatePaymentOrderAsync|Renew\(|DifferenceCount|LogWarning" \
  src/Platform/PlatformBilling src/Hosts/BitzOrcas.JobHost -g '*.cs'

# AutoRenewal and GraceSweep have dedicated tests; reconciliation, adapters, and endpoints do not.
rg -n "PaymentGatewayAdapter|AutoRenewalJobExecutor|GraceSweepJobExecutor|ReconciliationJobExecutor|PaymentEndpointGroup" \
  tests -g '*.cs'

Back to Billing · Payments and callbacks · Invoices and idempotency