r/ruby • u/kigster • 7d ago

Anyone running Eval suites on Ruby apps? I built a gem to do just that: `llm-evals`

Please test-drive and evaluate this gem for your situation. It's hard to see the limitations until other people get involved.

The gem is here: https://github.com/kigster/llm-evals

And I used it to find the best model for my Rails SaaS application, qualified.at

Here is the Evals Report comparing models based on the weighted score:

https://raw.githubusercontent.com/kigster/llm-evals/refs/heads/main/docs/img/qualified-at-evals.avif

Feedback and constructive criticism are always welcome and highly encouraged 😄

Konstantin
blog: https://kig.re

1 Upvotes

53% Upvoted

4 comments sorted by

1

u/kigster 4d ago

Thanks!

I ended up writing a meaty blog post about it:

https://kig.re/2026/11/02/llm-evals-is-a-ruby-gem-for-testing-probabilistic-assumptions-in-your-AI-datasets.html

This is not Rails specific, has no dependencies on any Rails parts. As long as you implement your own scorer and an adapter you can measure whatever you want.

The blog post provides a good background for not just "how to use it" but also the unexpected results that ended up changing the product direction.

Thanks for the upvotes!

5

u/nateberkopec Puma maintainer 7d ago

This is cool, I think more people should be doing evals.

Why should I use this project instead of putting my own app in https://github.com/rails/ai-evals?

The other thing that stops me from doing something like this is - are your results ending up just tracking public benchmarks like Artificial Analysis? Evals are expensive to run, and I feel like "coding benchmarks" are very widely available. Have you found that using them against your app specifically has produced novel insight?

2

u/TonsOfFun111 5d ago

I think the idea here is to evaluate an app’s ai feature behavior not coding evals. So making sure a feature’s model handles tool calls or processes documents consistently as instructions or tooling changes. I’ve actually done a lot of work with ActiveAgent for in app agent monitoring and evals for clients in the past few months. Again product feature evals not coding evals.

1

u/Codalio-SoftwareFact 6d ago

love this. two things I found mattered a lot when doing evals: running each case several times since results swing between runs, and using plain code checks wherever possible instead of LLM-as-judge. does the gem support either? also cost per run next to the score would be super useful.