RiskControl testing and production operations

A risk-control test must prove more than “this score maps to this level.” Release evidence must show that signals are trustworthy, challenges really block, tickets cannot be stolen, tenant evidence stays isolated, and every failure policy matches the consumer’s business risk.

1. Evidence that exists today

A fake Captcha proves that LoginFlow checks Passed. It cannot prove Redis GETDEL, ChallengeId binding, TTL, or the provider implementation.

2. Score boundary matrix

Test each lower bound, upper bound, and adjacent value:

Also enumerate all 16 hit combinations of the four built-in factors. Assert total score, factor code, Evidence, level, and challenge together. This test makes the default policy’s unreachable Block branch visible instead of silently assuming it is exercised.

3. Captcha one-time-use contract

[Theory]
[InlineData(CaptchaType.ImageCode)]
[InlineData(CaptchaType.Slider)]
[InlineData(CaptchaType.PointSelect)]
public async Task Ticket_Should_Be_Consumed_Atomically(CaptchaType type)
{
    var provider = fixture.Resolve(type);
    var challenge = await fixture.GenerateKnownAnswerAsync(
        provider,
        tenantId: "tenant-a",
        sessionId: "session-a");

    // The first correct answer succeeds and atomically consumes the Redis value.
    var first = await provider.VerifyAsync(
        new CaptchaVerifyRequest(challenge.Id, challenge.Answer),
        CancellationToken.None);
    first.IsSuccess.Should().BeTrue();
    first.Value!.Passed.Should().BeTrue();

    // Replaying the same ticket cannot pass, even with the correct answer.
    var replay = await provider.VerifyAsync(
        new CaptchaVerifyRequest(challenge.Id, challenge.Answer),
        CancellationToken.None);
    replay.IsSuccess.Should().BeTrue();
    replay.Value!.Passed.Should().BeFalse();
}

Production providers use randomness. Inject a controllable challenge source or decode RenderData in the test. Replacing the provider with a fake and calling the result a provider contract test leaves the security mechanism untested.

4. Context-theft tests

At minimum, prove:

• tenant-a/session-a cannot be used as tenant-b;
• session-a cannot be used as session-b within the same tenant;
• a Login ticket cannot be used for password reset or contact submission;
• a prefix mismatch does not call GETDEL, so the legitimate holder can still use the ticket;
• overlong IDs, control characters, and malformed segments are rejected;
• username normalization matches pseudo-session derivation.

[Fact]
public async Task Filter_Should_Reject_Passed_False()
{
    provider.VerifyAsync(
            Arg.Any<CaptchaVerifyRequest>(),
            Arg.Any<CancellationToken>())
        .Returns(Result.Success(new CaptchaVerification(false, "mismatch")));

    // The current filter calls next; the repaired contract returns 400 instead.
    // Exercise a real mapped endpoint so this also verifies filter attachment.
    var response = await client.PostAsJsonAsync(
        "/api/protected-form",
        new
        {
            captchaChallengeId = "captcha:tenant-a:session-a:n1",
            captchaAnswer = "wrong",
        });

    response.StatusCode.Should().Be(HttpStatusCode.BadRequest);
    protectedHandler.CallCount.Should().Be(0);
}

Run this as an HTTP test against a real endpoint. A direct unit call cannot prove endpoint-generation and filter-attachment behavior.

5. Behavior provider contracts

Use a local stub HTTP server to cover:

1. Aliyun and Tencent success, rejection, and malformed JSON;
2. non-2xx, connection failure, DNS error, timeout, and caller cancellation;
3. invalid Provider configuration failing startup instead of silently selecting Aliyun;
4. Secret, Ticket, and Randstr absent from logs, trace URLs, and error descriptions;
5. Tencent receiving the client Randstr and real user IP;
6. response-size, Content-Type, and redirect limits;
7. SDK URL restricted to an HTTPS allowlist.

An external-provider failure should reject that challenge. If the product chooses to fall back to ImageCode, the application must issue a new challenge explicitly; it must not reinterpret a consumed Behavior ticket.

6. Dual-ORM and tenant contract

Insert tenant-a and tenant-b assessments with the same username and timestamp into a real database. Run the same contract suite through EF Core and SqlSugar:

• tenant-a sees only tenant-a;
• User, Level, time, and pagination predicates are pushed down;
• soft-deleted facts are invisible by default;
• damaged FactorsJson produces a stable Failure;
• formal operate-as uses Effective Tenant, not the operator’s current-user TenantId;
• Host-wide analysis uses a separate permission and audit port.

The store’s explicit tenant predicate is still required even when the ORM has a global filter. Defense in depth is part of the store contract.

7. Fault-injection matrix

The two fail-open paths are not equivalent: Login removes the challenge, while risk-driven MFA falls back to TOTP. Dashboards must preserve that distinction.

8. Metrics and logging

Recommended low-cardinality metrics:

• risk_assessment_total{level,challenge,source};
• risk_engine_failure_total{stage};
• risk_policy_fallback_total{reason};
• captcha_generated_total{type,provider};
• captcha_verified_total{type,outcome};
• captcha_ticket_store_failure_total{operation};
• risk_assessment_persistence_failure_total;
• p50/p95/p99 latency for assessment, each factor, and each provider.

Do not place TenantId, UserId, IP, or ChallengeId in metric labels. Put only necessary redacted versions in access-controlled structured logs. Never log answers, ticket-store values, vendor secrets, or full device information.

9. Triage path

Start with the assessment and consumer branch before rotating secrets or modifying policy. Otherwise an implementation bug can be mistaken for an attack-pattern change.

10. Pre-release drills

1. disable Redis and confirm readiness prevents protected traffic;
2. inject invalid policy JSON and confirm fallback alerting plus strict-tenant handling;
3. time out the Behavior provider and confirm request denial without secret leakage;
4. make the assessment database unavailable and confirm alerting or compensation;
5. use a tenant-a challenge from tenant-b and confirm the valid ticket is not consumed;
6. submit a wrong answer to a filter-protected endpoint and confirm the handler does not run;
7. query a target tenant as Host operate-as and verify permission, data plane, and audit agree.

11. GA gate

• real provider and filter tests run in CI;
• every protected endpoint appears in an attachment inventory and an architecture test;
• ChallengeId binds tenant, session, and scenario; wrong context does not consume it;
• vendor secrets are absent from URLs, and provider/scene/Randstr/IP contracts are validated;
• engine, policy, and persistence fallbacks have metrics, alerts, and tenant-level policy;
• dual ORM, operate-as, Evidence redaction, and retention pass integration tests;
• the UI shows built-in factor reachability and makes the default unreachable Block threshold explicit.

Back: Integration boundaries · Back to RiskControl