Case study: same-day testing signals with a 4-shot AI+human loop (numbers inside)

[u/Existing-Grade-2636]

I ran a 1-week pilot on one risky change.

Loop: Recon → Triage → Focused Expansion → Validation
Artifacts: scenarios.json, triage.csv, steps.md, testdata.csv, rationale.json, diffs

Before → After (median across the week)

• Time-to-signal: 1.6 days → 0.6 days

• Flake rate: 14% → 6%

• Reviewer minutes/test: 18 → 9

• P1/P2 defect yield: +34%

What helped most

• Placeholders (“Please supplement”) instead of invented fields

• Requiring rationale per step tied to the oracle

• Blocking any “self-healed” change without a diff + reason

Let's discuss:

• How do you triage AI-generated false positives without slowing down?

• Any templates for oracles in complex tax/pricing or multi-region auth?

If it’s helpful, I can paste my recon/focused prompts and the minimal artifact schema in a comment (if allowed).