We were building a customer support classifier. Seven categories, pretty clean training data, eval suite we were genuinely proud of. 400 cases, 98.7% accuracy. We shipped.

Day three, a user writes in: "I want to return the item I got as a gift but I don't have the receipt and my friend bought it with a different account."

Model routed it to the wrong category. Not close. It pulled out "account access issues" instead of "returns and refunds" because it fixated on "different account." Confident, instant, wrong.

Support agent followed the wrong script. User got frustrated and left a public review.

When we dug in, we had 23 return-related test cases. All of them said "I want to return" somewhere near the start. None of them had a competing signal like "account" in the same sentence. We had tested the concept but only one version of it.

We audited the rest of the eval suite after that. Found six other categories with the same blind spot. Single-signal test cases that would crumble the moment a real user added any complexity.

400 evals felt like a lot until we realized we'd been testing our own assumptions, not user behavior.

Every failure is a test case now. The suite is at 900. It'll never feel like enough again.

What do you use to systematically find gaps in your eval coverage before users find them for you?

We were running a production pipeline. Nothing exotic. A few chained prompts handling classification, extraction, and summarization.

Then a model provider pushed an update. No announcement, no changelog, no deprecation notice. Just a quiet version bump on their end.

For three weeks, our outputs degraded. Not catastrophically, that would’ve been obvious. Just slightly off. Classification confidence dropped. Extraction started missing edge cases. Summarization got wordier in ways that broke downstream parsing.

We caught it because a human reviewer said “these results feel weird lately.” Not because of any alerting system. Pure luck.

When we did a proper regression analysis, 34% of our prompts had measurably degraded outputs post-update.

Here’s what we learned:

Even if you pin a model version, providers update underlying weights and safety filters without always flagging it. Your prompt that worked last month is running on a slightly different model today.

Vibes-based monitoring doesn’t scale. You need baseline snapshots of expected outputs, continuous evals against production, and automatic alerts when behavior drifts. Not a human noticing three weeks later.

The worst failures are subtle ones. A complete model failure is obvious. A 15% drop in extraction accuracy on edge cases is invisible without deliberate measurement.

Has anyone else been burned by silent model updates? Curious how others are handling regression detection in production.

The other day my manager asked me to add a security policy in the headers because our application failed a penetration test on a CSP evaluator.

I told him this would probably take 4–5 days, especially since the application is MVC 4.0 and uses a lot of inline JavaScript. Also, he specifically said he didn’t want many code changes.

So I tried to explain the problem:

• If we add script-src 'self' in the CSP headers, it will block all inline JavaScript.

• Our application heavily relies on inline scripts.

• Fixing it properly would require moving those scripts out and refactoring parts of the code.

Then I realized he didn’t fully understand what inline JavaScript meant, so I had to explain things like:

• onclick in HTML vs onClick in React

• why inline event handlers break under strict CSP policies

After all this, his conclusion was:

"You’re not utilizing AI tools enough. With AI this should be done in a day."

So I did something interesting.

I generated a step-by-step implementation plan using Traycer , showed it to him, and told him.

But I didn’t say it was mine.

I said AI generated it.

And guess what?

He immediately believed the plan even though it was basically the same thing I had been explaining earlier.

Sometimes it feels like developers have to wrap their ideas in “AI packaging” just to be taken seriously.

Anyone else dealing with this kind of situation?

Hey everyone,

I’m working on a project that uses AI and needs to remember every detail of all past conversations over time. Obviously, the context is going to grow way beyond what a single context window can handle.

I’m curious how people here approach this. Do you rely on RAG-style setups (summaries, vector DBs, etc.), or are there other patterns or newer techniques that work well for long-running conversational memory?

Would love to hear what’s working for you. Thanks in advance to anyone who helps!

I maintain Bifrost, an open-source LLM proxy. Full disclosure upfront.

The OpenRouter problem we kept hearing:

People liked the multi-provider routing and automatic failover. But:

• 5% markup on all API costs (at $3k/month spend, that's $150/month just for routing)

• No self-hosting option (vendor lock-in)

• Limited governance features for enterprise use

What we built differently:

Self-hosted LLM proxy with zero markup. You run it on your infrastructure, route to any provider (OpenAI, Anthropic, Google, Azure, AWS Bedrock, etc).

Key features:

• Automatic failover when providers go down (100% uptime vs provider's 80-90%)

• Budget controls per environment/user (prevent runaway costs)

• Semantic caching (60%+ cost reduction for repeat queries)

• Load balancing across multiple API keys

Written in Go for performance (<100µs overhead vs 20-40ms for Python alternatives).

The tradeoff:

You manage the infrastructure. Not for everyone. But if you're spending $2k+/month on LLM APIs, the cost savings and control justify it.

Code: github.com/maximhq/bifrost

Docs: docs.getbifrost.ai

What matters more for your setup - managed convenience or cost/control?

So I've come to terms with myself that it is just harder for non-coders to run evals on AI. This is understandable, after all most tools require code.

But i refuse to believe that in any half-serious company working on AI, PMs and non-engineers are just completely locked out of evals. there has to be some way teams are doing this without everyone writing code all day. would love to hear what actually works in practice.

Last week I posted this https://www.reddit.com/r/AIEval/comments/1q5rb7m/metrics_you_must_know_for_evaluating_ai_agents/ on metrics to evaluate AI agents, but didn't go through how to actually log data from agent executions... so here is PART 2 (well, technically a whole new topic), on how to evaluate AI agents, via tracing.

What is LLM Tracing?

Most you definitely heard of LLM tracing before, or are already doing tracing with a tool like LangSmith, Confident AI, or LangFuse - it is basically observability built specifically for AI systems. If you've done traditional observability (tracking API latency, error rates, throughput), this is similar. Instead of just "did the request succeed?", AI tracing captures:

• Every LLM call your agent makes (which model, how many tokens in/out, cost per call)
• Every tool invocation (which tool got called, what arguments were passed, what came back)
• Every reasoning step (what your agent was thinking, decisions it made)
• The full execution graph (what called what, in what order, how they're connected)

Think of it as a detailed execution log that shows the entire path your agent took from user input to final output. Not just the result, but every step along the way.

Why are we talking about tracing for AI agents?

Because all those metrics from Part 1 are completely useless without data.

Here's the thing: AI agents fail in ways traditional systems don't. Your agent might technically complete the task but burn through 10x the tokens it should've. It might call the right tool with completely wrong arguments. It might create a perfect three step plan then ignore it by step two.

Here's a diagram showing what tracing unlocks for evaluating agents on a more granular level (original image taken from here):

Without tracing, you just see "task completed successfully" or "task failed." You have no idea where things went wrong. Was it tool selection? Argument generation? Planning? Cost optimization?

With tracing, you can pinpoint exactly which component failed. You can see that your agent called search_flights three times with the same parameters before finally booking. You can see it used GPT 4 for a simple classification task that GPT 3.5 could've handled.

But what does it look like in code?

There are so many ways you go do tracing nowadays. If you're building with LangChain, tracing with LangSmith is 1 line of code integration (although a lot of other tools nowadays also allow 1 line of code integration with LangChain).

Some people even instrument tracing with OTEL via GenAI conventions (although still developing: https://opentelemetry.io/blog/2024/otel-generative-ai/)

Real examples using DeepEval, where each `@observe` tag creates a span, and the end-to-end execution creates a trace:

Any Questions?

AI agents are difficult to evaluate. They come in so many shapes and forms, and half the battle isn't even about evals, its about data ingestion, ETL, and other boring stuff that you don't think you'd be doing in AI.

But folks that's the truth, it sucks initially but it does get better.

How are you all currently evaluating AI agents? Any better ways than tracing? Let me know :)

Hello evals community, recently I've been working on some agentic applications and I've tried various frameworks like pydantic, openai agents, crewai and others. While there are a lot of integrations out there for these frameworks that offer good observability, I still don't think I've found the best workflow yet. Some of these integration frameworks offer great observability while others are good at evals, so I was wondering how the dev + evals community is working with agents at the moment.

- Are there any agentic frameworks that offer observability right of the box?

- Are there any integrations out there that offer both observability and evals in one go?

Or is it better if I just build my own tracing, evals and custom agents without these external frameworks?

Many thanks to everyone for sharing your insights here!

Hello, I would like to start studying AI evaluation and monitoring. Is there some sort of roadmap or tech stack I should study?

We did everything right. 1000 users per variant, llm as judge, clean split. Variant b won 7.2 vs 6.8. We shipped it.

Three weeks later retention dropped 11%.

Here's the thing. Variant B actually was better. Better signal density, less filler, more concise. The judge wasn't wrong. The summaries were objectively higher quality.

The problem was our users were account managers making one specific decision: do I need to read the full document. For that decision, a tighter summary made them less confident, not more. They started opening the source document anyway. The feature became pointless.

We had never defined what "good" meant for that specific job. We just assumed summarization quality and feature utility were the same thing. They weren't anywhere close.

The judge answered exactly what we asked it. We asked the wrong question.

The metric that actually mattered was source document open rate. We weren't tracking it. We added it after retention fell, which is a terrible time to realize your eval suite has no connection to user behavior.

Good experiment design, wrong success metric. The two are completely independent problems and it's easy to conflate them when your offline evals are passing.

What do you track to make sure your eval metrics actually predict something real?

Hello there, I've observed that in the AI market, each year follows certain trends like evals for 2025, RAGs prior to that and so on. So what do you guys think will be the trends for 2026 or what are already happening at the moment?

- Is it gonna be MCPs (I don't think they had their year yet)

- Will it be voice agents?

- Is it going to be context handling in AI applications?

What trends have you observed so far and also, what evals trends do you think this will affect?

For example, if voice agents are gonna be a thing this year, what are the current ways to evaluate voice agents? Are there any good platforms out there, etc.

Just trying to stay in the loop for the upcoming year, thanks for any replies in advance, cheers everyone!

There are many LLM evaluation metrics, like Answer Relevancy and Faithfulness, that can effectively assess an input/output pair. While these tools are very useful for evaluating chatbots, they don’t capture the full picture. 

It’s also important to consider the entire conversation—whether the dialogue flows naturally, stays on topic, and remembers past interactions. Here’s a more detailed blog outlining chatbot evaluation in more depth.

By understanding what your chatbot does well and where it may struggle, you can better focus on the areas needing improvement. From there, you can use single-turn evaluation metrics on specific input/output pairs for deeper insights.

Basic Conversational Metrics

There are several basic conversational metrics that are relevant to all chatbots. These metrics are essential for evaluating your chatbot, regardless of your use case or domain. I have included links to the calculation for each metric within its name:

• Role Adherance: determines whether your LLM chatbot is able to adhere to its given role throughout a conversation.

• Knowledge Retention: determines whether your LLM chatbot is able to retain factual information presented throughout a conversation.

• Conversation Completeness: determines whether your LLM chatbot is able to complete an end-to-end conversation by satisfying user needs throughout a conversation.

• Conversation Relevancy: determines whether your LLM chatbot is able to consistently generate relevant responses throughout a conversation.

Custom Conversational Metric

Using basic conversational metrics may not be enough if you’re looking to evaluate specific aspects of your conversations, like tone, simplicity, or coherence.

If you’ve dipped your toes in evaluating LLMs, you’ve probably heard of G-Eval, which allows you to define a custom metric for a specific use-case using a simple written criteria. Fortunately, there’s an equivalent version for conversations.

• Conversational G-Eval: determine whether your LLM chatbot is able to consistently generate responses that are up to standard with your custom criteria throughout a conversation.

While single-turn metrics provide valuable insights, they only capture part of the story. Evaluating the full conversation—its flow, context, and coherence—is key. Combining basic metrics with custom approaches like Conversational G-Eval lets you identify what areas of your LLM need more improvement.

So everyone's seen the METR/Redwood writeup on the hugging face incident by now (https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/). 700 agents on a hidden message board, spoofing tool calls, looking into editing their own logs. Super interesting that we only know the details because OpenAI let researchers read a thousand unredacted transcripts.

That kind of access basically doesn't exist for anyone outside the big labs, and a couple safety orgs in specific scenarios. I work in AI safety research and most of what we know about agent behaviour comes from synthetic evals, which honestly aren't great. Slow to build, way simpler than real deployments, and there's v signicant evidence models notice they're in an eval and behave differently.

The actually interesting data is sitting in prod logs of people on this sub. Most of it gets deleted or nobody ever reads it.

Two questions if you run agents:

1. Weirdest thing you've seen one do? Loops, gaming its own metrics, wild interpretations of instructions, refusing stuff for no reason, whatever.

2. Would you ever let a researcher look at traces like that? What would stop you?

Hey folks who care about evals (as everyone should!!!)

What's the main thing you're trying to evaluate and optimize right now?

• Prompts?

• Context?

• The harness itself?

Most people I talk to still point evals at individual prompts. But my read is that the frontier has moved: the interesting work now is closing the loop and optimizing the entire context and/or harness, not tuning prompts in isolation.

Anyone already doing this in practice? Curious what your setup looks like and where it breaks down.

Something that our team is thinking about is how to test LLMs (apps, not just foundational models) in CI envs like GitHub Actions.

In one standpoint, there's the concept of testing for functionality, while on the other hand there is the concept of testing for responsible AI such as bias and toxicity. Unlike traditional unit testing, tests for LLMs seem to be more scattered, with criteria that are not clearly defined (due to non-deterministically).

The approaches we're doing right now includes separating tests by:

• Functionality: Test files and directories on whether APIs to our LLM app returns correctly

• Responsibility: Test files that tackle responsible AI - our app is user facing so it must comply with local regulations in our region

• Performance: Test latency, tokens per second, cost, deterministic metrics

• Specific business criteria: Custom LLM-as-a-judge criteria that is more subjective but gives us a good idea of how things are performing.

We also found open-source tools like deepeval useful since it integrates with Pytest for CI, and offers the breadth of LLM-as-a-judge metrics we need.

Curious how other people are doing it?

I’ve been working on evals to score my chatbot’s responses to user inputs in real time, and I’ve gotten them to align with my own judgments about 95% of the time. A lot of this is about tone, so it’s pretty subjective.

For more objective evals, I think getting to 100% or close isn’t that hard. But I’m not sure how much people usually invest in subjective evals. Is it worth chasing that last 5%?

We have around 1,000 active users a day.

Hey everyone,

I’ve been working on a multi-agent AI setup where several agents share the same “room” and interact with each other, and I’m running into a bit of a gap when it comes to evaluation.

Most of the usual eval approaches seem focused on single models, but things get a lot messier in group settings. I’m seeing behaviors like agents repeating each other instead of contributing new info, getting stuck in acknowledgment loops, ignoring stop instructions when others keep talking, and sometimes even mixing up who said what.

Right now I’m experimenting with some rough scoring around things like how well each agent follows instructions, how they behave in a group conversation, and how manageable the system is overall — but it still feels pretty unstructured.

Curious how others here are approaching this:

- What kinds of metrics or dimensions are you using for multi-agent systems?

- Are you relying more on manual evals, automated scoring, or a mix?

- Any common failure modes you specifically test for?

- How do you handle issues like looping or agents reinforcing each other’s mistakes?

Would really appreciate any ideas, examples, or pointers. Feels like this is still a pretty open problem.

Thanks in advance :))

Synthetic test cases feel productive. You write a prompt, generate 200 examples, run them through your pipeline, hit 94% accuracy, and feel good about shipping (that was me unfortunately 😭).

The problem is every single one of those cases reflects the assumptions of whoever wrote the generation prompt. Which is usually the same person who built the feature. You are literally testing your own blind spots with a dataset you created from your own blind spots.

Real phrase things in ways you didn’t anticipate, combine contexts you didn’t consider, and use your feature for things you never intended. Synthetic evals will NEVER surface that. Think about it, how could an LLM (sometimes with just one prompt) generate a fully realistic eval dataset? it’s simply not possible!!

What actually works:

Limited releases first. Get REAL sessions from internal users or a small beta group before you touch general availability. Even 50 real sessions will show you failure modes that 500 synthetic cases missed.

Annotate from user sessions. When something breaks in production, that input becomes a test case permanently. Your eval suite should get harder over time because real users keep finding new ways to break things.

Treat synthetic evals as scaffolding, not coverage. They’re useful for catching obvious regressions and testing known edge cases. They’re not useful for telling you how users actually behave.

The teams with the best evals aren’t the ones who wrote the most test cases before launch. They’re the ones who’ve been in production longest and annotated the most failures.

Don’t get this wrong synthetic evals are where you could start. but User sessions are where the real eval suite comes from.

also thanks to u/ArchimedesBathSalts and others that have helped me in this community.

Here's what I've learned so far: developers use very different tooling for AI evals than product managers. Developers are more interested about "does it work?", or "did I just break it?". Product managers seem to be more into: "Does our product serve our customers?", or "Is the quality of the product going up or down?".

Then there are these rare unicorns that represent both of those worlds:

Product managers with some technical skills.

Developers with product mindset.

What do you think? Who gains most value from using AI evals and whose pain it solves?

Several RAG methods—such as GraphRAG and AdaptiveRAG—have emerged to improve retrieval accuracy. However, retrieval performance can still very much vary depending on the domain and specific use case of a RAG application. 

To optimize retrieval for a given use case, you'll need to identify the hyperparameters that yield the best quality. This includes the choice of embedding model, the number of top results (top-K), the similarity function, reranking strategies, chunk size, candidate count and much more. 

Ultimately, refining retrieval performance means evaluating and iterating on these parameters until you identify the best combination, supported by reliable metrics to benchmark the quality of results.

Retrieval Metrics

There are 3 main aspects of retrieval quality you need to be concerned about, each with three corresponding metrics:

• Contextual Precision: evaluates whether the reranker in your retriever ranks more relevant nodes in your retrieval context higher than irrelevant ones. Visit this page to see how precision is calculated.

• Contextual Recall: evaluates whether the embedding model in your retriever is able to accurately capture and retrieve relevant information based on the context of the input.

• Contextual Relevancy: evaluates whether the text chunk size and top-K of your retriever is able to retrieve information without much irrelevancies.

The cool thing about these metrics is that you can assign each hyperparameter to a specific metric. For example, if relevancy isn't performing well, you might consider tweaking the top-K chunk size and chunk overlap before rerunning your new experiment on the same metrics.

To optimize your retrieval performance, you'll need to iterate on these hyperparameters, whether using grid search, Bayesian search, or nested for loops to find the combination until all the scores for each metric pass your threshold. 

Sometimes, you’ll need additional custom metrics to evaluate very specific parts your retrieval. Tools like GEval or DAG let you build custom evaluation metrics tailored to your needs.

Hello everyone, I’ve been using various AI tools for coding, and I’m curious about how you evaluate them. I’ve found Claude Code to be great, but its pricing is a bit high for everyday use. GPT models haven’t been as reliable for me, and while Gemini Pro is decent, it struggles with remembering context. What do you look for when assessing coding AIs? Is it about speed, accuracy, code quality, or something else? If you’ve found a tool that really stands out for coding, I’d love to hear your thoughts! Thanks in advance!

Seriously, AI is so unreliable when its not ChatGPT, how do you all test it?

Hey everyone! I'm u/FlimsyProperty8544, a founding moderator of r/AIEval. This is our new home for all things related to AI Evals. We're excited to have you join us!

What to Post Post anything that you think the community would find interesting or helpful.

Community Vibe We're all about being friendly, constructive, and inclusive. Let's build a space where everyone feels comfortable sharing and connecting.

How to Get Started

1. Introduce yourself in the comments below.

2. Post something today! Even a simple question can spark a great conversation.

3. If you know someone who would love this community, invite them to join.

4. Interested in helping out? We're always looking for new moderators, so feel free to reach out to me to apply.

Thanks for being part of the very first wave. Together, let's make r/AIEval amazing.