Step-Up Rollout and Operations

Every misconfiguration of re-authentication has two safe backstops: toggle off = zero behavior anywhere (the evaluator’s first exemption vote; endpoints fall back to plain RBAC) and AuditOnly = record without intercepting. Rollout can therefore proceed as “understand the blast radius first, then enforce”, and rollback is always a single step.

Rollout order

1. Enable the toggle: on the host “feature distribution” page, turn the tenant-scoped feature identity.stepup on. With no policy rows and no declared baselines the system still does nothing — the toggle only unlocks enforcement. Demonstration-scope deployments seed two pilot purposes (API-key and HMAC-client lifecycle commands) with audit-only rows as the rollout starting point; the seed creates only on a missing natural key and never reverts rows that have already progressed.
2. Observation period (AuditOnly): create an auditOnly policy row for the target purpose (see the quickstart). Traffic passes as before, but every “would have required step-up” request writes a StepUpAuditOnlyObserved security audit. On the security audit page filter Module = Identity and the action to answer three questions: daily hit volume, which audiences and hours concentrate it, and whether frontend coverage is sufficient (old clients fail hard on 403s).
3. Switch to required: once the hit volume is acceptable, flip the policy row to required. Saving invalidates the local snapshot immediately and broadcasts to every instance — no restarts.
4. Rollback: at any time switch the row back to auditOnly or off, or turn the toggle off. Policy rows are a data surface; rollback never loses configuration.

Threshold tuning guidance

All thresholds live in the StepUp section (defaults and constraints in the contract reference). Tune against observation data rather than copying defaults:

A special note on window vs one-shot: low-frequency high-severity operations (payments, permission changes, MFA changes) fit one-shot tickets (window 0) or short windows; high-frequency batch flows fit 300–900 s windows. A one-shot ticket burned by a business failure costs a re-verification — the price of compressing the replay window to zero.

Runbook essentials

The full rollout and operations runbook shipped with the pilot slice: docs/architecture/07-security/0707-step-up-runbook.md in the BitzOrcasVNext repository (Redis prerequisites, toggle steps, observation-period interpretation, the audit query surface behind five acceptance metrics, threshold tuning, offline identity verification as the lost-device fallback, and rollback-by-toggling-off); long-term mechanism decisions live in ADR 0706 of the same repository. Until those are mirrored publicly, this section is the operational authority:

• Change entry point: policy rows and control-plane settings are managed only through the admin pages. Editing the policy table directly in the database bypasses guardrail validation and snapshot-invalidation broadcast, and is prohibited.
• Diagnostics: on the security audit page, query Security records with Module = Identity; six actions (challenge issued, verify succeeded/failed, grant rejected, observation, policy changed) cover every decision path.
• Drills: rehearse fail-closed by stopping Redis, and rehearse the alert path by deleting a version key — in staging regularly, not on production first contact.

Troubleshooting Q&A

Q1: after enabling the toggle, every request returns 403 Identity.StepUp.PolicyUnavailable.

Fail-closed is working; this is not mis interception. The root cause is almost always an unreachable credential or policy store: Redis not deployed, wrong connection string, network partition. Fixing storage restores service. Note the designed semantics: on deployments without Redis, identity.stepup is locked off (pass-through = plain RBAC), while “toggle on but Redis lost mid-flight” rejects with policy-unavailable — the first prevents accidental enablement, the second prevents bypass. Both directions are intentional.

Q2: audits show GrantInvalid with reason=versionKeyMissing — what now?

The version-key TTL is computed as “window ceiling + challenge TTL + safety padding”, so under normal operation any live credential always has a live version key. Seeing this alert means the key was deleted externally (manual Redis cleanup, an eviction policy misfire, a shared instance across environments); validation fails closed and writes a high-severity audit. Action: audit your Redis cleanup jobs and exclusion rules so the {app}:{env}:v1:stepup: prefix is preserved; the credential itself needs no handling — the user just re-verifies.

Q3: what does a rising GrantPurposeMismatch audit count mean?

The credential is valid but its purpose differs. A valid credential binds a single purpose, so cross-purpose replay always lands here. One or two occasional hits usually trace to frontend concurrency queuing; a sustained rise is an attack signal: someone holds a valid credential and is trying it on other sensitive endpoints. Action: pull the subject and source IP from the audit, follow your incident process (revoke that user’s sessions; the credentials are already dead via versioning), and check frontend versions for old logic without purpose isolation.

Q4: the admin page rejects a save with PolicyConflict, but I cannot see the conflict?

PolicyConflict details are per-field: “the window exceeds the host ceiling of 900”, “the purpose is outside the host-configurable whitelist”, “mandatory purposes cannot be switched off or downgraded to observation” — fix the named field. “A policy row already exists for the same tenant, purpose and scope” means the logical scope sets are identical after normalization: edit the existing row. Version conflicts (PolicyVersionConflict) resolve by refreshing and retrying.

Q5: a policy row changed, but some instances still show the old behavior?

A write invalidates the local snapshot and broadcasts via LocalResourceSync to every instance; beyond the broadcast there is a fallback TTL (PolicySnapshotTtlSeconds, factory 60 s). If instances still disagree after a minute, check each instance’s subscription health on the sync channel.

Related topics

• Concept and architecture: the full fail-closed list and revocation semantics
• Contract reference: every configuration key and rate-limit tier
• Frontend integration: legacy-client behavior during rollout
• Production security checklist: general pre-launch checks

Diagnostic command cheat sheet

The fixed triage order: toggle and purpose hits first, the audit decision chain second, storage last.

# 1) Purposes that would demand step-up for the caller — an empty array means
#    the toggle is off or no row/baseline exists.
curl -s -H "Authorization: Bearer $TOKEN" \
  -X QUERY https://localhost:5001/api/identity/step-up/my-purposes

# 2) Decision preview for one purpose: isRequired=false means calls are not
#    intercepted — stop looking for storage-side causes.
curl -s -H "Authorization: Bearer $TOKEN" \
  https://localhost:5001/api/identity/step-up/factors/identity.user.remove

# 3) Policy rows and control-plane state: confirm window/factors/scope against
#    memory (guarding against out-of-band edits).
curl -s -H "Authorization: Bearer $ADMIN_TOKEN" \
  -X QUERY https://localhost:5001/api/identity/step-up/policies

Six security-audit actions cover the decision chain (Module = Identity, queryable on the security audit page):

Observation-mode audit envelope sample (other fields follow the unified audit envelope contract):

// reason carries machine-readable context; challenge ids are stored only as
// SHA-256 hashes, never in plaintext.
{
  "category": "Security",
  "module": "Identity",
  "resourceType": "StepUpPolicy",
  "resourceId": "identity.user.remove",
  "action": "StepUpAuditOnlyObserved",
  "result": "Success",
  "level": "Security"
}