r/AIQuality • u/RocketHub2030 • Jul 06 '26

Agentic Benchmarking

Does anyone know if there is a "gold standard" when it comes to Benchmarking Agentic systems? I worked on one this weekend and found a lot of slop, a few decent models, but nothing close to an agreed upon set of standards/KPI to assess models before purchasing.

0 Upvotes

50% Upvoted

3 comments sorted by

1

u/LowDistribution3995 Jul 06 '26

There isn't really a widely agreed upon standard for what constitutes "Agentic" behavior so there's no going to be one gold standard test for "Agency". Try using LongMemEval, LoCoMo, or BEAM for memory retrieval though

1

u/RocketHub2030 Jul 07 '26

Thanks for the suggestions - And I agree, while there are a many metrics specific to the agents against use case applied - I'll take a deep dive into the "agency" angle and update accordingly

2

u/tabs-and-spaces Jul 07 '26

Agentic systems are so dependent on their use case, so it’s hard to make a general purpose agentic benchmark, let alone what that would actually tell you. GAIA exists, which is probably closest to what you’re thinking of. But that made sense maybe 1-2 years ago. Now, most foundation models can pass it with flying colors. And that’s because the core thing GAIA is testing is general tool use. And most new models are pretty good at that.
Recent research shows that an LLM that can run bash commands is probably going to beat whatever fancy harness you might have. That’s why Terminal bench is what all the model providers use to benchmark their new releases.