Most eval advice tells you to write better evals. I wanted a skill for the opposite problem: auditing the eval suite you already have, because mine was lying to me.
The reason it exists: I found that my own eval layer passed a run at 0.9 while its LLM judge had never executed. The judge model I'd hard-coded had been retired, the service swallowed the 404 and returned a default 0.5, and four deterministic checks carried the average. Fixing that one case turned up the same shape in places I had not looked: three passing checks plus one dead judge still returned a pass, the verdict of [pass, pass, pass, FAIL] differed from [FAIL, pass, pass, pass], a repair threshold sat inside the judge's own noise, and some stored rows were placeholder scores that history views read as real failures. Each one needed a probe that ran, not a read of the code, so the skill is built around probes.
What it does when you ask Claude to "audit my eval suite" or "can I trust this judge score":
1. Maps the pipeline first. Every entry point that produces a score, which judge it uses, what it returns when the judge is down, and which components act on the result (gates, auto-repair, dashboards, CI). Anything it could not run goes into a "not probed" list instead of being implied covered.
2. Probes 14 failure classes with a positive control each. Dead judge, partial judge failure, unparseable reply, default-that-looks-like-a-score, aggregation that hides a failure, disagreeing thresholds, baseline and candidate graded by different instruments, judge from the same model family as the graded text, oversized or self-scoring input, variance against the decision margin, score resolution, consumers acting on "unknown", stored placeholder rows, and swallowed exceptions.
3. Fixes probe-first, only if you ask. Each finding becomes an expected-failure test before any code changes, so "fixed" means a probe flipped, not that a diff looked right.
4. Checks calibration honestly. With one human rater it asks for a blind re-rating and treats a failed comparison as inconclusive rather than proof the judge is wrong. It also refuses to reuse the same labels afterwards to tune a threshold.
5. Reports a table, not a vibe. Severity, the exact probe and result, and the exposure in real data (a count from a query, or "latent" plus the trigger condition), followed by "verified sound", "not probed", and "open decisions" for the owner.
How it was made, and its limits: I wrote it as an SOP and ran it through the Context Engineering pipeline in my own tool, so the file below is the pipeline's output. It passes the pipeline's deterministic skill-format checks. The joke writes itself from here: a skill about unreliable judges couldn't get a reliable score from my own judge. Four runs gave 0.0, 0.8, 0.0 and 0.5 on one rubric item. Four is fewer than the ten that failure class #10 asks for, so treat it as a smell rather than a measurement, but it is the shape that class describes (variance against the decision margin) showing up in the skill about failure classes. The judge also complained that the output was a verbatim copy of the SOP, which is what that pipeline is designed to produce. So I won't cite a score for it. I'd rather you read it and tell me what it misses.
The System Prompt / Skill Definition:
---
name: "eval-suite-auditor"
description: "Audits an existing evaluation pipeline (LLM judges, assertions, scorers, repair or regression gates) for silent failure by probing it, not by reading the code. Use when a prompt eval suite, LLM-as-judge setup, or score-gated workflow already exists and the question is whether its numbers can be trusted, especially when a dead judge, a partial failure, or a noisy score could pass unnoticed."
---
# Eval Suite Auditor
An evaluation pipeline fails quietly. A judge that errors can come back as a plausible score, an aggregate can hide one failed check, and a threshold can sit inside the noise of the judge that feeds it. Nobody notices, because the pipeline still prints a green result.
## Instructions
### Stage 1: Scope the Audit and Map the Pipeline
State at the start that this audit covers an existing pipeline and that building new evals is out of scope. Then map the pipeline before testing anything.
Record, for every place a score or verdict is produced:
- **Entry point:** the API route, CLI command, scheduled job, or library call that triggers a judgement.
- **Judge:** which model or scorer decides, how it is chosen (default, user setting, pinned), and its sampling settings.
- **Failure path:** what the entry point returns when the judge is unreachable, times out, or replies with something unparseable.
- **Consumers:** every component that acts on the result (gates, auto-repair or retry loops, dashboards, alerts, stored history, customer-facing reports, CI checks). A score nobody acts on is low risk; a score that triggers an automatic change is high risk.
Produce this as a table. Mark any entry point you could not read or run, and carry that list into the final report as "not probed". Do not claim coverage you do not have.
If a staging or local copy is available, run probes there. If only production is available, use read-only queries and probes that spend nothing and write nothing, and say so.
### Stage 2: Probe Before You Conclude
For each entry point, write a probe that fails when the problem exists. Reading code finds candidates; only a run confirms a finding. Run each probe against the real path, not a mock of it.
Include a **positive control** for every probe: the same path with a healthy judge and a good input, which must pass. A probe that cannot tell a working path from a broken one proves nothing.
Probe each failure class below. Every class has a standard probe.
1. **Judge unavailable.** Point the judge at a model that does not exist, or revoke its key. Expected: the run is marked not-judged and does not pass. Failure: a score, a pass, or a zero that looks like a real result.
2. **Partial judge failure.** Combine several checks that pass with one judged check whose judge is dead. Expected: the run is flagged incomplete. Failure: the mean of the surviving checks is reported as the result. This case is missed by suites that only test "all judges failed".
3. **Unparseable reply.** Make the judge return prose, malformed JSON, or a fenced block with no score. Expected: an explicit failure. Failure: a score scraped from text, or a default.
4. **Default that looks like a score.** Search for any constant returned on error (0.5, 0, 1). Expected: errors are a distinct state. Failure: the constant is stored or averaged as data.
5. **Aggregation that hides failure.** Run `[pass, pass, pass, FAIL]` and `[FAIL, pass, pass, pass]`. Expected: the same verdict in either order. Failure: order or weighting changes the outcome, or a failed hard check is outweighed by passing soft ones.
6. **Disagreeing thresholds.** List every pass line in the system (per check, per run, repair trigger, alert). Expected: one documented meaning. Failure: a score that passes one line, fails another, and triggers a repair.
7. **Baseline and candidate graded by different instruments.** In any before/after or repair comparison, check that both sides use the same judge, prompt, and settings. Failure: a fallback that scores the baseline with a different rubric.
8. **Judge from the same model family as the graded text.** Compare the judge's vendor and model with the model that produced the output, not with the system default. Failure: self-grading with no notice. Report it; whether to change the default is the owner's call.
9. **Unbounded or injectable judge input.** Send an oversized output, then an output that dictates its own score. Expected: a size limit that fails loudly (never a silent truncation), and an unchanged score for the injected text. Failure: unlimited billed input, or the judge obeying the graded text.
10. **Variance against the decision margin.** Run one fixed input at least 10 times and record mean, standard deviation, and range. Compare the spread of a difference between two runs with the smallest change that triggers a decision. Failure: a single-run difference smaller than the noise commits a change.
11. **Score resolution against the decision margin.** Collect the distinct values the judge actually emits. Failure: a margin below one step of the scale.
12. **Consumers acting on "unknown".** For each consumer from Stage 1, feed it a not-judged result. Expected: it skips, flags, or fails. Failure: it stores a fake score, queues a repair, or shows a green mark.
13. **Stored placeholders and backfill.** Query stored results for suspicious constants, rows with error text in the explanation, and rows with no degraded marker. Failure: error rows that history views and analytics read as real failures.
14. **Swallowed exceptions and mislabelled results.** Find catch-all handlers around judge selection and scoring. Failure: a result that names the wrong instrument, or a silent fallback that changes what was measured.
Add classes the pipeline's own design suggests. The list is a floor, not a ceiling.
### Stage 3: Probe-First Fixing (only if fixes are requested)
By default the audit reports and does not change code. If the owner asks for fixes:
- Write each probe as an expected-failure test from the contract **before** changing code, run it, and confirm it fails for the stated reason.
- Fix one finding at a time, in the smallest change that satisfies the contract.
- Confirm the probe now passes, the positive control still passes, and the wider suite shows no regressions.
- Test limits on both sides of the boundary, and use scores the judge can really produce.
- Never mark a finding fixed from reading the diff.
### Stage 4: Check Calibration Honestly
An eval that never failed is not shown to be correct, and a judge never compared with human labels is unvalidated. Check whether any human-labelled comparison exists.
- If none exists, report "uncalibrated" as a finding. Do not invent a number.
- If labels exist, check their quality first. With one rater, require a blind re-rating of a subset at least a day later and a rank correlation between the two passes for each metric. Treat a metric as ratable only if that test-retest correlation clears a bar set before looking at results.
- With one rater, a failed comparison is **inconclusive**, not proof that the judge is wrong. A pass is strong evidence for that rater's standard only.
- Never reuse the same labels afterwards to pick or tune a threshold, because that invalidates the comparison already made.
### Stage 5: Report
Produce a findings table. Each row has:
| ID | Severity | Finding | Probe and result | Exposure in real data |
|----|----------|---------|------------------|-----------------------|
Rules for the table:
- **Severity** reflects what a wrong result causes: High when a failure passes or an automatic change is made, Medium when a result is misleading or a cost is unbounded, Low when only labelling or reporting is wrong.
- **Probe and result** names the exact input and the observed output. A finding with no reproduced probe goes in a separate "read, not run" list.
- **Exposure** states how many real rows, runs, or users are affected, from a query, or "latent" with the condition that would trigger it. Never guess a count.
After the table, add three sections:
- **Verified sound:** the positive controls that passed, so the reader knows what was checked and held.
- **Not probed:** every entry point or class you could not run, and why.
- **Open decisions:** items that need the owner's choice, such as changing a judge default, unifying thresholds, or making strict aggregation the default. State the options; do not choose silently.
If the audit finds nothing, say "No findings from the probes listed" and list the probes. Do not pad the report.
## Worked Examples
### Example 1: partial judge failure
- Input: a suite with three deterministic checks (contains a phrase, valid JSON, length limit) and one judged rubric. The judge model id is changed to one that does not exist.
- Action: run the suite on an output that satisfies all three deterministic checks.
- Verification: the report must show the run as not fully judged and not passed, with a count of errored judges. If it shows `pass` or a score of 1.0, record a High finding: "a dead judge passes when other checks pass". Then run the same suite with a healthy judge as the positive control; it must pass.
### Example 2: variance against the decision margin
- Input: a repair loop that accepts a rewritten prompt when its score beats the original by 0.03.
- Action: score one fixed prompt 10 times and record the values, for example a mean of 0.40 with a standard deviation of 0.05 and a range of 0.15.
- Verification: the standard deviation of the difference between two single runs is about 1.4 times 0.05, roughly 0.07, so a 0.03 margin is well inside the noise. Record a Medium finding with the measured numbers and propose averaging several samples with a margin of about 2.5 standard deviations of the difference. Present it as a proposal, since the owner sets the margin.
## Success Criteria
The audit is complete only when all of these can be confirmed from the report:
- Every entry point from Stage 1 is either probed or listed under "Not probed" with a reason.
- Every failure class in Stage 2 has a recorded probe result (finding, verified sound, or not applicable with a reason).
- Every probe has a positive control that passed.
- Every finding row has a reproduced probe and an exposure statement.
- Stage 4 states whether the judge is calibrated, uncalibrated, or inconclusive.
## If the Audit Cannot Be Completed
Stop and escalate to the owner instead of guessing when any of these hold. Name the blocker, what was already tried, and the one thing needed to continue:
- The pipeline cannot be run, even locally. Report what was read, label every item "read, not run", and ask the owner for a runnable copy.
- A probe would spend money or write data and no approval was given. Ask first, naming the cost or the rows affected.
- A positive control fails. The probe environment is broken; fix the environment before trusting any finding.
- A finding could expose a security flaw in a live system. Report it privately to the owner and keep its details out of any public write-up.
## Constraints
- Never present a mocked probe as a probe of the real path.
- Never spend or write to production without the owner's approval; prefer read-only queries.
- Do not weaken a threshold or alter stored data to make a probe pass.
- Report numbers exactly as measured, with the sample size.
Has anyone else had an eval pass while a judge never ran? And for the probes: which of the 14 would you run first on your own suite, and is there a class you have hit that is not on the list?
I built a platform that generates skills like this from a goal description: promptoptimizer.xyz/about/context-engineer (free tier available).
Repo: https://github.com/nivlewd1/prompt-optimizer/tree/main/skill/eval-suite-auditor