Secret management is not complete when a value moves out of appsettings.json. Production needs ownership, workload access, rotation, revocation, compromise recovery, and an inventory of affected artifacts and tenants.
Classify by purpose
| Type | BitzOrcas example | Control plane |
|---|---|---|
| symmetric signing | JWT fallback, HMAC, Webhook | versioned Secret Store/KMS |
| asymmetric verification | license trusted public keys | integrity-controlled configuration |
| connection credential | SQL, Redis, RabbitMQ, S3 | workload identity + Secret Store |
| external provider | OAuth, SMS, email, AI key | one secret per provider |
| reversible app field | provider key, sensitive lead | purpose-specific Data Protection |
| validation credential | API Key, SCIM token | runtime hash, no plaintext retention |
A public verification key is not confidential, but unauthorized replacement is still a security failure.
Configuration is a delivery channel
JSON, user-secrets, environment variables, mounted files, and external providers deliver configuration; they do not automatically provide approval, rotation, or audit.
# Keep development material in user-secrets rather than the repository.dotnet user-secrets set \ --project src/Hosts/BitzOrcas.Api \ 'Jwt:Secret' '<development-only-secret-at-least-32-chars>'
# List local keys only in a private terminal; never copy values into CI logs.dotnet user-secrets list --project src/Hosts/BitzOrcas.ApiEnvironment variables may appear in container descriptions and diagnostics. Kubernetes Secret still needs etcd encryption, RBAC, audit, and restricted mounts.
JWT key rings
JWT startup requires issuer, audience, and signing material. Without Jwt:SigningKeys, Jwt:Secret must contain at least 32 characters. Missing material fails startup. Production should use versioned keys with kid.
{ "Jwt": { "Issuer": "bitzorcas-api", "Audience": "bitzorcas-client", "SigningKeys": { "2026-07": "<injected-current-key>", "2026-04": "<old-verification-key>" }, "ActiveKeyId": "2026-07" }}Teach every verifier the new key, switch issuance, wait for old tokens to expire, and then revoke the old key. Overwriting one secret immediately can invalidate every live token.
HMAC, API Key, and SCIM
HMAC clients need a recoverable secret to sign. API Key and SCIM validators need only a hash; current runtime maps SHA-256 hashes to trusted identity facts. The management plane should show plaintext only at creation.
create: CSPRNG plaintext → display once → save hash + prefixcall: receive plaintext → SHA-256 → fixed-time match → tenant/scopesrevoke: disable hash record → next request failsPrefixes such as boa_live_ aid identification but do not authorize. Static compatibility configuration is useful for tests, not a complete credential directory with owner, expiry, scope, use history, overlap, and revocation.
Webhook and provider secrets
The Webhook subscription repository encrypts secrets using IWebhookSecretCipher. Provider keys may use Data Protection for reversible field protection. Database ciphertext still depends on the outer key ring, and a compromised application identity may use the legitimate decrypt path.
Back up the database and key ring as a recoverable pair. Restrict application access and never log decrypted values.
Data Protection key ring
The API can persist the ASP.NET Core key ring to a fixed Redis key. Redis credentials, network policy, and key-ring data are sensitive controls. Never share one ring across dev/test/staging/production.
Purpose isolation remains mandatory even when one SaaS deployment shares an application ring across tenants. See Data Protection.
Runtime license material
Production/Staging runtime guards require signed licensing plus ProductId, ProductVersion, Environment, TenancyMode, DeploymentIdentityPath, CachePath, and at least one TrustedPublicKey.
The trusted public key may be deployed in controlled configuration. The private license-signing key must remain in the issuer’s KMS/HSM and never ship with the product. Identity and offline-cache files need stable, restricted storage.
Standard rotation
- create a version with owner, purpose, creation, and expiry metadata;
- prove consumers can read multiple versions;
- deploy new verification capability first;
- switch issuance or active connections;
- observe failures and old-version hits;
- wait through token, retry, and offline-job windows;
- revoke the old version and prove it cannot be used;
- update inventory and rehearsal records.
For databases and brokers, create a new account and cut traffic before revoking the old account. For JWT/Webhook, bound any dual-verification window with an explicit end time.
Application access pattern
Handlers depend on options or capability interfaces, not direct environment-variable reads. The composition root validates configuration and centralizes redaction.
// Bind and validate at startup; business handlers receive a capability.builder.Services .AddOptions<ConnectorOptions>() .BindConfiguration("Connectors:Payments") .ValidateDataAnnotations() .ValidateOnStart();
// Log a version identifier, never SecretValue.logger.LogInformation("Payment connector key version {KeyVersion}", options.KeyVersion);CompositeSecretStore priority and failure semantics
Infrastructure composes multiple ISecretStore implementations in DI order and visits only stores whose IsAvailable is true. Reads follow these rules:
| Current store result | Next action | Final chain result |
|---|---|---|
| Success | Return immediately; skip lower-priority stores | Successful value |
ErrorType.NotFound | Continue to a lower-priority store | Preserve SecretNotFound only when every available store reports NotFound |
| Any other failure | Log a warning and continue | NoStoreAvailable if no later store succeeds |
| Non-cancellation, non-OOM exception | Log a warning and continue | NoStoreAvailable if no later store succeeds |
| No available store | No read is possible | NoStoreAvailable |
A NotFound result is safe only when every available source confirms that the secret is absent; a caller may then use an explicit deployment default. If any source reports a store fault, unknown failure, or exception, the chain fails closed instead of disguising infrastructure failure as missing configuration. OperationCanceledException and OutOfMemoryException are not swallowed.
RotateSecretAsync also tries available stores in priority order and returns on the first success. The current write path does not preserve NotFound classification; if none succeeds, it returns NoStoreAvailable. Do not assume that a read fallback store is the rotation target. The deployment inventory must identify the Provider with write permission.
Compromise response
Treat a secret pushed to Git, CI logs, chat, tickets, or a public image as compromised. Revoke/rotate first, then investigate. Deleting the message or rewriting Git history does not make the old value safe.
Freeze the key, determine the exposure window, inspect use, rotate derived tokens, notify the owner, repair the channel, and add detection. For decryption keys, assess which historical data may have been read.
Prevent leakage
- scan commits, history, CI logs, images, artifacts, and source maps;
- redact ProblemDetails, structured logs, audit detail, and metric labels;
- configuration diagnostics report presence/source, never value;
- test fixtures use explicit test prefixes and independent values;
- treat screenshots and support bundles as sensitive artifacts;
- minimize runtime identities that can read each secret.
Release evidence
Every class needs owner, readers, version, rotation period, last rotation, revoke path, and recovery rehearsal. Critical paths need one dual-key rotation drill and one compromise-revocation drill.
Startup success proves only that today’s value works. Commercial GA evidence also covers store policy, Provider priority and failure injection, workload identity, scanning, runtime redaction, rotation, and recovery.