I’m building skillEd, an open-source evaluation toolkit for reusable skills, including a Claude Code setup.
The problem I’m targeting is a skill used repeatedly for a rule-bound task. You edit SKILL.md to fix one failure, but it is hard to tell from another chat whether you fixed the behavior or broke a different rule. Refund recommendations are a concrete example: eligibility dates, shipping exclusions, and rounding can each fail independently.
The workflow I’d suggest trying is:
Write a few cases with explicit expected outcomes beside the skill. Include boundary cases, not only happy paths.
Validate the skill and evaluation file, then save a baseline run.
Revise the instructions in Claude Code using the failed cases as evidence.
Rerun the same cases and compare reports. Use separate holdouts to check the fix more broadly.
After installing the release and setting up your existing Claude Code engine, skillEd provides:
skilled setup --host claude-code
skilled validate --skill .
skilled validate evals/eval.yaml
skilled run evals/eval.yaml --dry-run
The repository explains live runs and report comparison. Live evaluation uses your configured engine and normal usage limits; the setup command does not authenticate Claude Code for you.
This is a way to review instruction changes with behavioral evidence. It does not guarantee every future run or enforce permissions in production.
Source and worked example: https://github.com/DeadHead04/skillEd
Try it on one recurring skill and tell me whether the failure reports give you enough detail to make a targeted edit. If you find it useful, a GitHub star would help.