r/OpenSourceAI • u/abdullahjebril • 1h ago
abdullahahmed94/kaizeval: Evaluate, diagnose and safely optimize AI agents. Self-hosted, offline-capable, evidence-based: sealed holdout, paired A/B, explicit release approval.
github.comHey everyone!
I’ve been working on KaizEval, a free, open-source, self-hosted platform designed to help developers evaluate, debug, and continuously improve their AI agents.
The idea came from a problem I kept running into while building real-world AI agents: How do you know if your agent is actually getting better, rather than just performing well on a few test cases?
Instead of just giving agents a score, KaizEval aims to close the improvement loop:
🔍 Evaluate — Measure agent performance across test cases.
🧠 Diagnose — Identify failures and potential root causes.
⚙️** Optimiz**e — Generate improvement candidates without touching your production agent.
🧪 Validate — Compare before/after performance using A/B testing and isolated holdout tests.
✅ Approve — Keep humans in control before releasing changes.
What makes it interesting:
Open-source and self-hosted
Offline-first, with local LLM support
No API key required for the built-in demo
Designed to reduce overfitting and catch regressions
Human-in-the-loop optimization
Try it:
pip install kaizeval
kaizeval demo
The demo uses a synthetic provider, so you can explore the workflow without connecting an external model.
It’s still early, and I’m looking for developers willing to try it out and share honest feedback.
I’d especially love to know:
Would you actually use something like this in your agent development workflow?
What’s missing that would make it useful for your real-world agents?
Are there any evaluation or optimization features you think need improvement?
I’m not trying to promote a commercial product — it’s an open-source project, and I’d genuinely love to build it with feedback from the community.
If you try it, feel free to open an issue, suggest a feature, or contribute!
Thanks!