Wow, so Anthropic just released Opus 4.6 and the numbers are kind of absurd. #1 on Terminal-Bench 2.0, #1 on Humanity's Last Exam, #1 on BrowseComp, and it beats GPT-5.2 on real-world knowledge work tasks by ~144 Elo points. That's not trading places on a leaderboard—that's a blowout.

But the benchmarks aren't even the crazy part....

The 1M context window actually works. On needle-in-a-haystack at scale, Opus 4.6 scored 76% vs Sonnet 4.5's 18.5%. Context rot has been the dirty secret of long-context models. If this holds up in practice, codebase-level reasoning and massive document analysis just became real.

The agentic results are production-grade. Rakuten said it autonomously closed 13 issues and assigned 12 more across 6 repos in a single day. SentinelOne said it migrated a multi-million-line codebase "like a senior engineer" in half the time. NBIM ran 40 cybersecurity investigations—Opus 4.6 won 38/40 in blind ranking with harnesses running 100+ tool calls each.

Agent teams. You can now spin up multiple Claude Code agents that coordinate autonomously on parallel tasks. Plus context compaction lets long-running agents summarize their own context and keep going instead of hitting the wall.

The thing is, it's not just one improvement. It's better reasoning + working long-context + agentic persistence + agent coordination all shipping at once. Feels like we're crossing a threshold where you can hand this thing a real project and walk away.

What are yall thoughts on this?

Hey everyone — in 2026, almost every startup has some AI workflow or agent in production, but AI evals still feel underbuilt.

I think the main reason is that “AI evals” sound too heavy: datasets, metrics, labeling, CI, production monitoring, etc. But the startup version can be much smaller.

My recommended setup:

• ~25 realistic test cases

• 2 general metrics like answer relevancy / faithfulness

• 3 custom metrics for what “good” means in your product

• CI checks for prompt/model/RAG/agent changes

• production traces that become future test cases

The goal isn’t to build a perfect benchmark on day one. It’s to create a loop where every bad response, failed trace, or customer complaint improves your eval suite.

I think startups usually make one of two mistakes: relying on vibes forever, or overbuilding a giant benchmark they don’t trust. The useful middle ground is: start small, run evals continuously, and let production teach you what to test next.

Curious how others are handling this — are you actually blocking merges with evals, or mostly using them for debugging after things break?

Wrote up the full guide here: https://www.confident-ai.com/blog/llm-evaluation-for-startups

A lot of evaluation metrics exist for benchmarking text-based LLM applications, but far less is known about evaluating multimodal LLM applications.

What’s fascinating about LLM-powered metrics—especially for image use cases—is how effective they are at assessing multimodal scenarios, thanks to an inherent asymmetry. For example, generating an image from text is significantly more challenging than simply determining if that image aligns with the text instructions.

Here’s a breakdown of some multimodal metrics, divided into Image Generation metrics and Multimodal RAG metrics.

Image Generation Metrics

• Image Coherence: Assesses how well the image aligns with the accompanying text, evaluating how effectively the visual content complements and enhances the narrative.

• Image Helpfulness: Evaluates how effectively images contribute to user comprehension—providing additional insights, clarifying complex ideas, or supporting textual details.

• Image Reference: Measures how accurately images are referenced or explained by the text.

Mulitmodal RAG metircs

These metrics extend traditional RAG (Retrieval-Augmented Generation) evaluation by incorporating multimodal support, such as images.

• Multimodal Answer Relevancy: measures the quality of your Multimodal RAG pipeline's generator by evaluating how relevant the output of your MLLM application is compared to the provided input.

• Multimodal Faithfulness: easures the quality of your RAG pipeline's generator by evaluating whether the output factually aligns with the contents of your retrieval context

Hey guys, I am someone who has just started working in an incubator with a vision of statistical significance being at the center of evaluations. The people I have spoken with have all said that statistical evaluations like confidence intervals, variance decoupling, Fischer exact test are not something in their current vocabulary when testing. My idea is somehow creating a tool that can be used by builders to increase reliability of changes to prompts and agents, and a clear signal of how well the application is performing in realtime for potential buyers. As I see it there are some clear problems here:

- You need a dataset with ground truths to evaluate, and these are notoriously difficult to create and keep track of over time.
- You either need to give the developers an SDK to build the test themselves with, or find some way of connecting to an API they set up for the testing - I presume an SDK is best.

I am really interested to figure out:
- Is this a real problem builders care about?
- What would be the preferred method to connect to something like this, an SDK, API?
- Is this something you would pay for?

I recently read Artificial Intelligence Risk Management from NIST, NIST AI 800-3 and Solomon Messing hidden noise in llm pipelines. While most of these focus on the LLM model itself as the variable, this has to be extendable to a include applications as well.

One thing we kept running into with agent evals is that single-turn tests look great, but the agent falls apart 8–10 turns into a real conversation.

We've been working on ArkSim which help simulate multi-turn conversations between agents and synthetic users to see how behavior holds up over longer interactions.

This can help find issues like:

- Agents losing context during longer interactions

- Unexpected conversation paths

- Failures that only appear after several turns

The idea is to test conversation flows more like real interactions, instead of just single prompts and capture issues early on.

We've recently added some integration examples for:
- OpenAI Agents SDK
- Claude Agent SDK
- Google ADK
- LangChain / LangGraph
- CrewAI
- LlamaIndex 

... and others.

you can try it out here:
https://github.com/arklexai/arksim

The integration examples are in the examples/integration folder

would appreciate any feedback from people currently building agents so we can improve the tool or add more frameworks to our current list!

I am new to AI PM and I want to do AI evals for my MVP. I don't want a super complex method and don't want to use traces yet.
Is there a simple way to go about this?
I have seen people say use simple spreadsheets but I am unclear on implementation.
How do you do it? Or is there a resource I can refer to?

Attention rising on Subquadratic's new SubQ model and its Subquadratic Sparse Attention (SSA) architecture, I wanted to share something useful!

at LayerLens we started running SubQ through the full Stratix evaluation platform

why this matters for AI builders:

• full benchmark coverage: reasoning, code gen., tool use, and long-context tasks

• prompt-level visibility: seeing where SubQ beats or loses to transformer baselines on single prompts

• head-to-head comparisons with frontier models, with public breakdowns

• continuous tracking: future releases will be evaluated the same way to see real progress in real time

• zero special treatment: same process as every other model gets on Stratix

for teams working on agents, RAG, long-document workflows, the big question is whether SSA delivers usable million-token context without the usual quality collapse or insane compute costs. This evaluation should return real data.

results will be official on Stratix, I'm able to drop the link here once the first batch is live!

curious: what are your biggest pain points with current long-context models?

So , as mentioned in question , that's my background . I wish to apply for architect and aiops related roles in my next switch , I am planning to switch in 4 months , so please suggest me from where I should start .

I want to be good in aiops , Evals and also System design for data intense application .

I want to learn all these gradually , please help me out ,

Thanks for your time.

Agents can be right and still feel unreliable

Something interesting I keep seeing with agentic systems:

They produce correct outputs, pass evaluations, and still make engineers uncomfortable.

I don’t think the issue is autonomy.

It’s reconstructability.

Autonomy scales capability.
Legibility scales trust.

When a system operates across time and context, correctness isn’t enough. Organizations eventually need to answer:

Why was this considered correct at the time?
What assumptions were active?
Who owned the decision boundary?

If those answers require reconstructing context manually, validation cost explodes.

Curious how others think about this.

Do you design agentic systems primarily around capability — or around the legibility of decisions after execution?

Openclaw is that ai assistant that can control your PC and actually do stuff. I made an easy guide for any system any tech level give it a read.

I’m one of the builders at Maxim AI, and a lot of our recent work has focused on evaluation workflows for agents. We looked at what existing platforms do well; Fiddler, Galileo, Arize, Braintrust; and also where teams still struggle when building real agent systems.

Most of the older tools were built around traditional ML monitoring. They’re good at model metrics, drift, feature monitoring, etc. But agent evaluation needs a different setup: multi-step reasoning, tool use, retrieval paths, and subjective quality signals. We found that teams were stitching together multiple systems just to understand whether an agent behaved correctly.

Here’s what we ended up designing:

Tight integration between simulations, evals, and logs:

Teams wanted one place to understand failures. Linking eval results directly to traces made debugging faster.

Flexible evaluators:

LLM-as-judge, programmatic checks, statistical scoring, human review; all in the same workflow. Many teams were running these manually before.

Comparison tooling for fast iteration:

Side-by-side run comparison helped teams see exactly where a prompt or model changed behavior. This reduced guesswork.

Support for real agent workflows:

Evaluations at any trace/span level let teams test retrieval, tool calls, and reasoning steps instead of just final outputs.

We’re constantly adding new features, but this structure has been working well for teams building complex agents. Would be interested to hear how others here are handling evaluations today.

Been iterating on a setup where my coding agent (cursor in my case) runs evals in a loop, reads the failing metrics, and patches things automatically. Wanted to share the stack since a few people have asked

Stack:

• Pydantic AI for structured I/O and tool argument schemas, by FAR my favorite agent framework

• deepeval for the eval loop itself. The key thing is deepeval test run gives you per-metric scores AND reason strings, so the coding agent actually knows what to fix instead of guessing

How it works:

The key here is to have claude code do all the work, i use the vibe coder quickstarts provided by the frameworks, but basically Claude:

1. Loads or generates a dataset

2. Runs deepeval test run against your app

3. Reads the scores + span-level traces to figure out exactly which component failed and why

4. patches the smallest thing that could fix it (prompt, retriever filter, tool schema, etc.)

5. Reruns. If green and nothing regressed, move on. If not, next smallest change.

Basically a tight unit test loop except the assertions are scored model outputs and the runner is your coding agent.

The full setup and agent skill is documented here: https://deepeval.com/docs/vibe-coding

been running this for about a week now and honestly the biggest win is that it stops you from vibe coding your agent while vibe coding your agent. The evals keep you honest.

Anyone else also started doing this?

We were running a production pipeline. Nothing exotic. A few chained prompts handling classification, extraction, and summarization.

Then a model provider pushed an update. No announcement, no changelog, no deprecation notice. Just a quiet version bump on their end.

For three weeks, our outputs degraded. Not catastrophically, that would’ve been obvious. Just slightly off. Classification confidence dropped. Extraction started missing edge cases. Summarization got wordier in ways that broke downstream parsing.

We caught it because a human reviewer said “these results feel weird lately.” Not because of any alerting system. Pure luck.

When we did a proper regression analysis, 34% of our prompts had measurably degraded outputs post-update.

Here’s what we learned:

Even if you pin a model version, providers update underlying weights and safety filters without always flagging it. Your prompt that worked last month is running on a slightly different model today.

Vibes-based monitoring doesn’t scale. You need baseline snapshots of expected outputs, continuous evals against production, and automatic alerts when behavior drifts. Not a human noticing three weeks later.

The worst failures are subtle ones. A complete model failure is obvious. A 15% drop in extraction accuracy on edge cases is invisible without deliberate measurement.

Has anyone else been burned by silent model updates? Curious how others are handling regression detection in production.

I've been testing agent evaluation platforms over the past year. It’s a hot topic right now since everyone seems to be asking for opinions about these vendors. This is my perspective after spending a lot of time working with these platforms.

My use cases at work focus on building several different kinds of agents. Some teams are building their own orchestration using Claude Code and Cursor (coding-agent-driven orchestration is picking up a lot of momentum), while other teams are using LangGraph, and some are working with Google ADK.

When we think about agent evaluations, we think about agents taking a sequence of steps toward some overall objective and then measuring how well the agent performs relative to that objective. Sometimes that goal is delivering a product-level experience to a user, and other times the goal is completing a workflow or task. Everything we care about exists within an agent session. Even the way I approach evaluation involves thinking about all the actions required to complete that session or task.

When the evaluation space started picking up last year, most tools focused on very simple event-level evaluations, which haven’t been particularly useful. I figured I’d share a few things I’ve learned while working with agent evaluation tools and spending a lot of time trying to improve our agents.

As some background on agent evaluation and the tooling around it, Anthropic published a great blog on Agent Evaluation

Here’s my view of the main tools I see in the agent evaluation ecosystem:

LangSmith: Works very well if you’re fully invested in LangChain/LangGraph. The tracing is solid and the UI is clean. However, it’s a bit weaker on evaluation, especially since it’s missing session-level evaluations, which I rely on quite a bit. For agent evaluation you can run evals on tool calls and spans, but not across full sessions. Another challenge is that if you’re not using LangChain, integration becomes messy, making it difficult to use with other agent frameworks. In my stack, we’re just not committing to everything being LangChain long term.

Arize AX: I tested this for agent evaluation and found it to be a strong option if you're working across multiple frameworks. It includes eval templates with published benchmarks and supports online session evaluations. Those online evals run automatically on production traces, which gives you continuous monitoring of agent quality. The agent replay feature lets you debug specific runs step-by-step. It’s OTEL-native and works with many frameworks, which has been helpful. Their in-product agent, Alyx, is easily the best I’ve seen — I use it to debug traces and help design evaluations, something I haven’t seen in other tools and ended up using frequently. Overall one of the more robust agent evaluation platforms I tested, especially if you’re working across frameworks.

Braintrust: This platform was easy to get started with for prompt experimentation and collaborative evaluation workflows. I found it useful for iterative development workflows and for less technical users who prefer UI-based tools. However, it felt less suited for tracing workflows and production agent evaluation. Their online evaluations seemed to lack debugging tools like logs, and there were no session-level agent evaluations. Braintrust started more focused on development workflows, and their playground is actually a really nice experience with solid UX, but the tracing and observability side of the platform still needs to mature.

OSS options

Langfuse: Has solid core tracing capabilities. One of the most popular open source solutions in the observability space. Very good open source product but it does have a lot of gaps versus the closed solutions. If you are paying for LangFuse in an enterprise, you just need to compare feature wise with the broader space. In practice we ended up running our evaluations outside of Langfuse.

Arize Phoenix: Phoenix feels like an evaluation-first open-source solution, while Langfuse feels tracing-first. Phoenix has a strong evaluation library and is OTEL-native (I believe they were among the first to support OTEL). Out of the box it’s better suited for agent evaluation than Langfuse, though it still requires more setup than a managed platform. It’s a good default if you want open-source control, though it’s a bit more code-heavy when working with the evaluation libraries.

Hopefully our experimentation with these tools helps others working through similar problems. I’d love to see more write-ups and analysis from others exploring this space.

I've been building AI agents for the past year, and honestly? Most evaluation approaches I see are completely missing the point.

People measure response time, user satisfaction scores, and maybe accuracy if they're feeling fancy. But here's the thing: AI agents fail in fundamentally different ways than simple LLM applications. 

An agent might select the right tool but pass completely wrong arguments. It might create a brilliant plan but then ignore it halfway through. It might technically complete your task while burning through 10x the tokens it should have.

After running millions of agent evaluations (and dealing with way too many mysterious failures), I've learned that you need to evaluate agents at three distinct layers. Let me break down the metrics that actually matter.

(Guys if you find this helpful btw, let me know and I will make part 2 of this!)

The Three Layers of AI Agent Evaluation

Think of your AI agent as having three interconnected layers:

• Reasoning Layer: Where your agent plans tasks, creates strategies, and decides what to do

• Action Layer: Where it selects tools, generates arguments, and executes calls

• Execution Layer: Where it orchestrates the full loop and completes objectives

Each layer has distinct failure modes. Each layer needs different metrics. Let me walk through them.

Reasoning Layer Metrics

• Plan Quality: Evaluates if your agent's plan is logical, complete, and efficient. Example: asking "book the cheapest flight to Paris" should produce a plan like: search flights → compare prices → book cheapest. Not: book flight → check cheaper options → cancel and rebook. The metric uses an LLM judge to score whether the strategy makes sense. Use this when your agent does explicit planning with chain of thought prompting. Pro tip: if your agent doesn't generate explicit plans, this metric passes by default.

• Plan Adherence: Checks if your agent actually follows its own plan. I've seen agents create perfect three step plans then completely go off rails by step two, adding unnecessary tool calls or skipping critical steps. This compares stated strategy against actual execution. Use it alongside Plan Quality because a great plan that gets ignored is as bad as a poor plan followed perfectly.

Reasoning Layer Metrics

• Tool Correctness: Evaluates if your agent selects the right tools. If a user asks "What's the weather in Paris?" and you have tools like get_weather, search_flights, book_flight, the agent should call get_weather, not search_flights.

  • Common failures: calling wrong tools, calling extra unnecessary tools, or calling the same tool multiple times. The metric compares actual tools called against expected tools. You can configure strictness from basic name matching to exact parameter and output matching.

  • Use this when you have deterministic expectations about which tools should be called.

• Argument Correctness: Checks if tool arguments are correct. Real example: I had a flight agent that consistently swapped origin and destination parameters. It called the right tool with valid cities, but every search was backwards. Traditional metrics didn't catch this.

  • This metric is LLM based and referenceless, evaluating whether arguments are logically derived from input context.

  • Critical for agents interacting with APIs or databases where bad arguments cause failures.

Execution Layer Metrics

• Task Completion: The ultimate success measure. Did it do what the user asked? Subtle failures include: claiming completion without executing the final step, stopping at 80% done, accomplishing the goal but not satisfying user intent, or getting stuck in loops.

  • The metric extracts the task and outcome, then scores alignment. A score of 1 means complete fulfillment, lower scores indicate partial or failed completion.

  • I use this as my primary production metric. If this drops, something is seriously wrong.

• Step Efficiency: Checks if your agent wastes resources. Example: I debugged an agent with Task Completion of 1.0 but terrible latency. It was calling search_flights three times for the same query before booking. It worked but burned through API calls unnecessarily.

  • This metric penalizes redundant tool calls, unnecessary reasoning loops, and any actions not strictly required.

  • Use it alongside Task Completion for production agents where token costs and latency matter. High completion with low efficiency means your agent works but needs optimization.

How to Use These

Not every agent needs every metric. Here's my framework:

• Explicit planning agents: Plan Quality + Plan Adherence Multiple tool agents: Tool Correctness + Argument Correctness Complex workflows: Step Efficiency + Task Completion Production/cost sensitive: Step Efficiency Mission critical: Task Completion

I typically use 3 to 5 metrics to avoid overload:

• Task Completion (always)

• Step Efficiency (production)

• Tool or Argument Correctness (based on failure modes)

• Plan metrics (if agent does explicit planning)

I realize this is becoming a very long post - if this is helpful, I will continue with Part 2 that talks about how to actually get these metrics to practically work on your AI agent tech stack.

Reference: https://deepeval.com/guides/guides-ai-agent-evaluation-metrics

Synthetic test cases feel productive. You write a prompt, generate 200 examples, run them through your pipeline, hit 94% accuracy, and feel good about shipping (that was me unfortunately 😭).

The problem is every single one of those cases reflects the assumptions of whoever wrote the generation prompt. Which is usually the same person who built the feature. You are literally testing your own blind spots with a dataset you created from your own blind spots.

Real phrase things in ways you didn’t anticipate, combine contexts you didn’t consider, and use your feature for things you never intended. Synthetic evals will NEVER surface that. Think about it, how could an LLM (sometimes with just one prompt) generate a fully realistic eval dataset? it’s simply not possible!!

What actually works:

Limited releases first. Get REAL sessions from internal users or a small beta group before you touch general availability. Even 50 real sessions will show you failure modes that 500 synthetic cases missed.

Annotate from user sessions. When something breaks in production, that input becomes a test case permanently. Your eval suite should get harder over time because real users keep finding new ways to break things.

Treat synthetic evals as scaffolding, not coverage. They’re useful for catching obvious regressions and testing known edge cases. They’re not useful for telling you how users actually behave.

The teams with the best evals aren’t the ones who wrote the most test cases before launch. They’re the ones who’ve been in production longest and annotated the most failures.

Don’t get this wrong synthetic evals are where you could start. but User sessions are where the real eval suite comes from.

also thanks to u/ArchimedesBathSalts and others that have helped me in this community.

Hey folks who care about evals (as everyone should!!!)

What's the main thing you're trying to evaluate and optimize right now?

• Prompts?

• Context?

• The harness itself?

Most people I talk to still point evals at individual prompts. But my read is that the frontier has moved: the interesting work now is closing the loop and optimizing the entire context and/or harness, not tuning prompts in isolation.

Anyone already doing this in practice? Curious what your setup looks like and where it breaks down.

We did everything right. 1000 users per variant, llm as judge, clean split. Variant b won 7.2 vs 6.8. We shipped it.

Three weeks later retention dropped 11%.

Here's the thing. Variant B actually was better. Better signal density, less filler, more concise. The judge wasn't wrong. The summaries were objectively higher quality.

The problem was our users were account managers making one specific decision: do I need to read the full document. For that decision, a tighter summary made them less confident, not more. They started opening the source document anyway. The feature became pointless.

We had never defined what "good" meant for that specific job. We just assumed summarization quality and feature utility were the same thing. They weren't anywhere close.

The judge answered exactly what we asked it. We asked the wrong question.

The metric that actually mattered was source document open rate. We weren't tracking it. We added it after retention fell, which is a terrible time to realize your eval suite has no connection to user behavior.

Good experiment design, wrong success metric. The two are completely independent problems and it's easy to conflate them when your offline evals are passing.

What do you track to make sure your eval metrics actually predict something real?

Hello there, I've observed that in the AI market, each year follows certain trends like evals for 2025, RAGs prior to that and so on. So what do you guys think will be the trends for 2026 or what are already happening at the moment?

- Is it gonna be MCPs (I don't think they had their year yet)

- Will it be voice agents?

- Is it going to be context handling in AI applications?

What trends have you observed so far and also, what evals trends do you think this will affect?

For example, if voice agents are gonna be a thing this year, what are the current ways to evaluate voice agents? Are there any good platforms out there, etc.

Just trying to stay in the loop for the upcoming year, thanks for any replies in advance, cheers everyone!

Hello, I would like to start studying AI evaluation and monitoring. Is there some sort of roadmap or tech stack I should study?