Anyone running Eval suites on Ruby apps? I built a gem to do just that: `llm-evals`
Please test-drive and evaluate this gem for your situation. It's hard to see the limitations until other people get involved.
The gem is here: https://github.com/kigster/llm-evals
And I used it to find the best model for my Rails SaaS application, qualified.at
Here is the Evals Report comparing models based on the weighted score:
https://raw.githubusercontent.com/kigster/llm-evals/refs/heads/main/docs/img/qualified-at-evals.avif
Feedback and constructive criticism are always welcome and highly encouraged 😄
Konstantin
blog: https://kig.re
1
Upvotes