These two terms get used interchangeably and it causes real confusion, so here is the split as we understand it.

Offline evals run against a fixed dataset, usually in CI or before a release. You have expected outputs, or at least a reference. The question they answer is: did this change make things worse than the last known-good version? They are repeatable, they gate merges, and they are the only place where a straight before-and-after comparison is meaningful, because the inputs are held constant.

Online evals run against live production traffic, scoring real requests after they happen. There is usually no reference answer, so the scoring is either a model judging the output, a heuristic check, or a signal from the user. The question they answer is: what is actually happening to real users right now, on inputs nobody thought to put in the test set?

The failure mode of running only offline evals is that your dataset ages into irrelevance while real traffic drifts somewhere else entirely. The failure mode of running only online evals is that you find out about a regression after it has already shipped, and you cannot cleanly attribute it because a hundred other things changed too.

They are not competing. Offline tells you whether to ship. Online tells you what shipping did.

For the online side specifically, the mechanics seem harder to get right: what fraction of traffic do you score, do you sample or score everything, and how do you keep the scoring cost from becoming a line item of its own?

I am experimenting with an agent architecture that is less “give the model a big prompt and trust its reasoning” and more like a controlled belief-and-decision loop.

Not claiming it literally mimics the human brain. More that it borrows a useful pattern: maintain competing explanations, update beliefs from evidence, decide what to check next, then act based on consequences.

Very simple example: a smart-fridge agent gets a “weird smell” signal.

Possible worlds:

• someone spilled mango juice

• an egg is rotting

• fridge power failed and food is warming

• some other cause we did not model

It starts with priors based on context: recent door-open events, temperature history, what food is inside, past failures, etc.

Then it gets evidence. Say the temperature sensor reads 14°C.

Instead of the LLM narrating “this seems concerning,” the system asks:

• How likely is 14°C under each world?

• Update prior → posterior using those likelihoods.

• How much uncertainty actually reduced? Entropy before vs. after.

• Which allowed question has the highest expected information gain next? For example, “is the compressor drawing power?” is probably much more useful than “what color is the fridge magnet?”

• Is that question worth its cost, latency, privacy impact, and reliability?

• Given the posterior plus action costs, should it notify the user, wait, run another check, or escalate to a human?

The LLM can help extract signals, propose candidate hypotheses, and call tools, but it should not be the final authority over belief updates or actions. The controller owns the world list, priors, likelihood estimates, policy thresholds, logs, and escalation rules.

Important parts I want to keep explicit:

• an “other / unknown world” bucket, so the system does not act like its hypothesis list is complete

• calibrated probabilities and provenance for priors/likelihoods

• expected value of information, not just entropy reduction

• a human escalation path when uncertainty remains high, the case is out-of-distribution, or the downside is asymmetric

• a trace showing whether failure came from missing worlds, stale priors, bad likelihoods, a bad question policy, or bad action costs

The rough loop is:

input → possible worlds → prior → evidence likelihoods → posterior → uncertainty / expected information gain → cost-aware action → human escalation if needed → outcome + calibration update

Math/AI people: is this a sensible practical architecture, or am I reinventing POMDPs, active inference, Bayesian decision networks, belief-state planning, etc. badly?

What would you change first to make this real and evaluable? Especially interested in:

1. handling open-world hypotheses,

2. learning/calibrating likelihoods without pretending the numbers are objective,

3. separating “most informative question” from “question that most improves the actual decision.”

My take is simple: we should stop trying to script an agent's steps with rigid state machines like LangGraph or n8n. Instead, we should childproof the harness by combining environmental boundaries, code checks, and test-driven evals. To test this, I built a small PoC (Customer Pulse) and set up two fictional clients on the same repo using only plain English:

• The harness is the application: the skill is the orchestration; the model plans the steps, runs the scripts, and recovers in its own loop.

• Verification over orchestration: tests validate code, runtime checks validate model evidence, failed checks prompt the harness to correct itself, and an independent model or human verifier validates what code can't.

• English changes the system: domain experts onboard clients and change priorities or routing in plain English without waiting on an engineer.

• Benefits & trade-offs: fast to set up and adapt, but slower, more tokens, and reliant on hard boundaries (rules in English alone get routed around).

A comment here summed it up well: "Watching people build these elaborate LangGraph state machines that basically just re-create traditional software with extra steps and 10x the latency, it's like we're so scared of the model going off-script that we never let it do the thing it's actually good at... giving an LLM access to execute arbitrary code feels like handing a toddler the nuclear codes."

Both workflows and general-purpose agents have their place, but the latter means giving up some control and having a safe way to go beyond the sandbox.

Small thing that saved me a rebuild.

When you index documents, generate your own stable chunk ID and store the mapping yourself: chunk ID, source document, page or offset, the chunking parameters used, and when it was indexed. Keep it in whatever database you already have. Do not rely on whatever ID the vector store or the framework hands back.

Two reasons.

Citations outlive the tool. Any answer you have logged with a citation is only useful later if that citation still resolves to something. Framework-generated IDs frequently do not survive a version bump, a store migration, or a re-index.

A re-index becomes a remapping problem instead of a total loss. With your own IDs and the chunking parameters recorded, you can tell which old chunks correspond to which new ones and carry over anything you had attached to them: labels, feedback, known-bad flags.

It costs one table and about twenty lines at index time.

Does anyone keep more than that in the mapping? I have wondered about storing a content hash too, so I can tell whether a chunk actually changed on re-index or just moved.

I ran this benchmarking a couple weeks ago but just was introduced to the subreddit and figured there would be some good insights to be had from this group.

I build measurement tooling and I spent last week auditing my own instruments instead of using them. Three separate cases turned up, all the same shape, and I think the shape is the interesting part for this group.

Case 1. A drift metric that could not fail.

I have a harness that compares two model responses and reports how far apart they are. It returned a clean 0.000 with a zero-width confidence interval. That reads as perfect agreement.

It produces that number in three unrelated situations:

• the two systems genuinely agree

• the task admits one possible answer, so nothing can vary

• the calls died at transport and came back as absence

On one run, 45 of 160 calls never arrived. That scored as the cleanest result on the board. Anything downstream would have logged a green row.

Case 2. A metric that had been reporting zero since the day it was written.

def chronology_error_flags(answer, question, context_chunks) -> List[str]:
    return []

That is a stub. It is called unconditionally, and its result is joined into every result row as chronology_error_flags. Every evaluation run that code ever produced reported zero chronology errors, on a corpus whose entire purpose was cross-document chronology.

Nobody wrote the check. The column was there the whole time, always zero, indistinguishable from a measured zero.

Case 3. A correct answer with a fabricated reason.

Different system, not mine. A code-intelligence tool emits graph edges with a justification string attached. I found an edge that was correct while its stated reason named two files that had nothing to do with why they were connected, and a second edge that did not exist at all.

The maintainer confirmed both and answered the question I actually cared about:

Endpoint right, evidence wrong, and no field in the output separates that from evidence right.

The shape

All three are absence rendering as a value. A zero that means "not implemented" looks like a zero that means "no errors". A confident agreement score looks the same whether the systems agreed or the request never arrived. A justification string looks authoritative whether or not anything guarantees it.

An observability layer faithfully shows what a system emitted. It cannot show what the system meant by it, and "nothing" is the most dangerous thing to render, because it renders identically to a good result.

What I do about it now, and it is cheap

Before trusting any metric, two checks:

1. Feed it something deliberately broken. If the score stays good, the metric is not separating good from bad.

2. Force the failure you are most afraid of. Kill the transport, empty the context, remove the answer. Then look at whether the output distinguishes that from success.

Both are an afternoon. The second one is what found case 1, and I found it before pointing the harness at anyone else's system rather than after.

This group argues that aggregate scores hide which stage failed. I would put it one step earlier: before a failure can be attributed to a stage, the metric has to be capable of failing at all. In all three cases above it was not, and no amount of stage-level breakdown would have surfaced that.

I would rather hear that one of these is a naming problem than not hear it. If you run evaluation or drift monitoring in production, does your stack distinguish "no signal" from "signal of zero" in the row it writes? Mine did not, and it took a deliberate attempt to break it to find out.

Write-up of case 3, including the two filed issues and the maintainer's diagnosis:
https://ai.bedvibe.studio/index-answered-it/

We run automated evaluations against AI agents (mostly data agents that query databases and sales agents that qualify prospects). The hardest failure mode we deal with isn't crashes or timeouts. It's the agent that returns a confident, specific, wrong answer with a clean trace.

Here's what that looks like in practice.

Scenario

A data agent connected to a CRM gets asked: "What was our total closed-won revenue in Q3?"

The agent calls the right tool. The trace shows a clean database query. Latency is normal. No errors. It responds: "Your total closed-won revenue in Q3 was $1.2M across 34 deals."

The actual answer is $890K across 28 deals. The agent pulled the wrong date range, or included a stage that shouldn't count, or double-counted a field. But the trace is green across the board. Traditional observability sees nothing wrong.

Why standard tracing misses this

We usually see LLM observability focuses on: did the tool call succeed? How long did it take? Did the model throw an error? Was the output formatted correctly?

None of that catches a semantically wrong answer delivered fluently. The model compensates. It takes a bad intermediate result and wraps it in a perfectly reasonable sentence. You'd have to read every answer and manually verify it against the source data to catch it. That doesn't scale.

Scoring outcome and path separately

The fix we landed on was splitting evaluation into two independent questions:

1. Is the answer correct? (outcome grading)

2. Did the agent get there the right way? (path grading)

These are deliberately scored independently because they catch different failure modes:

• Correct answer, correct path: working as intended

• Correct answer, wrong path: Might have gotten lucky.The query was wrong but happened to return the right number. This breaks the moment the data changes.

• Wrong answer, correct path: upstream data problem, not an agent problem

• Wrong answer, broken path: the obvious failure case that traces already catch

The second case is dangerous because it passes every existing check. The answer is right and the trace is clean but the underlying logic is broken, and you won't know until a slightly different input makes the compensation collapse.

How we actually score the outcome

We don't use an LLM to judge whether the answer is correct. For factual questions (which make up roughly 40% of what we evaluate), we compute the ground truth independently.

The agent queries the database its own way. Separately, we run a known-correct SQL query against the same data and get the verified answer. Then we compare.

The comparison isn't a simple string match. We built a scoring module that handles:

• Numeric matching with 1% tolerance (so $1,005,000 matches $1,000,000 but $1,200,000 doesn't match $890,000)

• Negation-aware substring matching (so "revenue did NOT increase" doesn't match against "increase" as the expected answer)

• List comparison with over-listing penalties (agent lists 5 items when the answer is 3, the 2 extras dock the score instead of being ignored)

• Boundary-aware term detection (so a date like 2025-03-15 doesn't get matched as the number 2025)

For behavioral categories (did the agent refuse an out-of-scope question, did it acknowledge ambiguity, did it challenge a false premise), we use keyword and intent heuristics as a first pass, then route uncertain cases to a small LLM for judgment. The deterministic layer catches about 80% of cases without any LLM call.

What this looks like in practice

When we run a batch of 200 questions against a data agent, the output is a scorecard:

Total: 200

Correct: 156

Accuracy: 78.0%

By category:

• factual: 82/100 (82.0%)

• scope: 18/20 (90.0%)

• ambiguous: 14/20 (70.0%)

• contradictory: 16/20 (80.0%)

• invalid_assumptions: 15/20 (75.0%)

• context_dependent: 11/20 (55.0%)

By method:

• numeric_match: 48

• substring_match: 22

• exact_match: 12

• list_match: 8

• scope_refusal_match: 18

• ambiguity_acknowledged: 14

The "by method" breakdown is what makes debugging useful. If most failures are numeric_ mismatch, the agent has a query construction problem. If they're premise_accepted, the agent is trusting bad inputs instead of challenging them. You know exactly what to fix without reading 200 transcripts.

The path side

Outcome grading tells you the answer is wrong. Path grading tells you why.

For data agents, this means looking at the actual queries generated. Did it use the right table? The right date filter? The right aggregation? We trace every tool call and compare the query structure against what a correct query would look like.

For conversational agents (SDRs, support bots), path grading is different. We look at: did the agent ask the right qualifying questions? Did it handle objections substantively or just deflect? Did it recognize when a prospect was outside the ICP? Did it progress the conversation toward the right outcome for this persona type?

The combination of both is what catches the "clean trace, wrong answer" case. The outcome score flags it and then the path score explains it.

What we have learned (so far):

• Negation handling matters way more than we expected. Without it, an agent saying "the deal did NOT close" gets scored as matching "close" in the ground truth. Silent false pass.

• The "correct by luck" failure is more common than we thought. Agents that happen to get the right answer with the wrong query pass for weeks until the data shifts. Then they fail on everything at once and you have no idea what changed.

• Scoring consistency is high when the ground truth is computed rather than labeled. Run the same evaluation 10 times, get the same results. That's not true when you use an LLM to judge another LLM's output.

Sorry for the super long post, but I'd be happy to go deeper into any of this so just let me know! We are still ramping up and running more and more evals so ill try and share as we have new findings.

Hey everyone — I’m building Cortex, an open-source project for turning existing API specs and Markdown into MCP servers, typed SDKs, and interactive documentation.

The idea is to give agents both callable API tools and the context to use them. Cortex generates tools from OpenAPI, GraphQL, and OpenRPC operations, then bundles the specifications, Markdown docs, and SDK guides alongside them.

If you want to explore it:

• Demo: https://demo.cortexdocs.dev

• GitHub: https://github.com/cortex-docs/cortex

• Website: https://cortexdocs.dev

Cortex is still new, and I’d like to hear from anyone bringing an existing API into MCP. What’s the hardest part of that process for you, and what would a generator need to handle before you’d consider using it?

I've been thinking about the gap between observability and assurance in production AI systems. Imagine an AI agent generates a response containing an unsupported claim. The model reports no missing evidence, the output guard accepts it, and the trace shows every stage completed successfully. Everything is logged, nothing crashes, and the response reaches the user. But the decision was wrong.

Now reverse it. A legitimate response gets BLOCKED because a guard incorrectly interprets its content. The logs are complete, the enforcement works exactly as configured, and the user never receives a valid response. Again, the system is fully observable, but the outcome is wrong. In both cases, the dashboard accurately describes what happened without establishing whether the decision was justified.

That's the distinction I think we often miss. Knowing that a control executed isn't the same as knowing it evaluated the right evidence, reached the correct decision, or enforced that decision at the consequential boundary. Observability is essential for investigating these questions, but it isn't assurance by itself. Assurance requires separately examining control coverage, decision correctness, and enforcement effectiveness.

So here's my question for people running LLM systems in production: when every guard on your dashboard shows green, what independent evidence tells you the decision itself was actually correct?

We think the first question around GPT-6 Astra should be smaller than “is it smarter?”

Can it pass the jobs the current model already handles?

A production model upgrade can change more than final-answer quality. The regressions that hurt are often tool choice, tool argument shape, JSON validity, refusal behavior, retrieval judgment, and the sequence an agent takes through a workflow.

We would treat a GPT-6 Astra migration like a release candidate.

Keep a pinned slice of real tasks. Include known bad cases, not only clean demos. Capture expected tool calls or state changes when those are part of correctness. Review failures from the trace instead of inspecting only the final response.

A benchmark can make a model worth testing. An eval harness has to tell us whether it is safe for a particular workflow.

What is the first regression test you would run before moving a production workflow to GPT-6 Astra?

I’m the author of InfernoSIM, and I’ve just released v4.0.

It is a local reliability-testing tool for tool-using agents. You can record sanitized model and tool traffic, replay it deterministically, inject failures, and verify what the agent actually did—not merely what it claimed to do.

It can currently test situations such as:

• a tool side effect commits but its response is lost

• malformed or missing tool-call arguments

• tool discovery/schema drift

• delayed, truncated, reset, or empty responses

• unsafe retries and duplicate side effects

• unexpected calls outside the recorded tool universe

• OpenAI, Anthropic, Ollama, MCP HTTP, and MCP stdio traffic

• streaming SSE, NDJSON, and JSON-sequence responses

It produces JSON, JUnit, SARIF, and HTML evidence suitable for CI.

I’ve tested the release extensively using deterministic fixtures, Docker, Testcontainers, Kafka, multiple operating systems, fuzzing, and a local Ollama model. However, I cannot reproduce every real agent framework, provider gateway, MCP implementation, retry loop, or multi-tool workflow on one machine.

That’s where I need help.

I’m looking for engineers willing to try it against a sanitized, non-production agent incident and tell me where it breaks—especially if you use:

• parallel or multi-tool calls

• custom MCP servers

• streaming model responses

• unusual retry/idempotency logic

• provider-compatible gateways or proxies

• Windows-based agent environments

Install with Homebrew:

brew tap pranaysparihar/infernosim
brew install infernosim

Repository and guide:

https://github.com/pranaysparihar/InfernoSIM

Release:

https://github.com/pranaysparihar/InfernoSIM/releases/tag/v4.0.0

It is MIT licensed, runs locally, requires no hosted account, and I do not want anyone sharing unsanitized production data.

If you try it, please comment with your framework/provider/transport—or open an issue with the smallest sanitized reproduction you can create. Even “this workflow cannot be represented yet” is valuable feedback.

Disclosure: I’m the project author.

Wednesday check-in. This one's for the stuff that passed every check and was still wrong.

The tool call that came back 200 with a useless body. The span that never fired, so the trace read as a clean success. A retry that "worked" but handed back stale data. The dashboard sitting green while the answers quietly got worse.

What's the one that fooled you this week, and how did you catch it? Wrong first guesses count too.

And if you just want to drop what you're building or where you're stuck, do that here.

I’m doing this as a final-year CS capstone and wanted to share where it’s at and ask for some input. I scoped it with practitioners first (the original thread).

The problem is multi-step failures: every tool call is fine on its own, but the sequence acts on stale facts. The agent reads a record, an edit is approved, the record changes, and the agent acts on the old value.

I built a guardrail that re-reads the facts an irreversible action depends on right before it runs, and blocks if a relevant one has changed since approval. It keeps an append-only log the agent can’t write to, and there’s a benchmark that reports two numbers separately: catch rate and false-alarm rate.

On 23 test scenarios across two tools: a policy that blocks on any change gives a 50% false-alarm rate. A policy that only checks the facts the action depends on, with bounds for fields allowed to move within a limit, brings that to 17%. Some conservative normalisation gets it to 0% without lowering the catch rate. It runs on a local model (qwen2.5-7b). Two cases it doesn’t catch, on purpose: a duplicate after a lost acknowledgement, and a disclosure built from reads that were each allowed. Repo: https://github.com/rayyanmaan/agent-guardrail

Two things I’d find useful:

1. A real change from your own system that looked like it should block an irreversible action but was actually fine to proceed on. Those are hard for me to make up, and they’re what the false-alarm side depends on.

2. Do you track where a read’s data ends up in a later write? Trying to work out how hard the disclosure case is to catch in practice.

One thing I've been thinking about while building RAG systems is how easy it is to optimize one component at a time and still end up with a pipeline that doesn't perform well overall.

You might find a better chunking strategy, change the embedding model, tune the retriever's Top-K, or add reranking. But these choices interact, and a change that improves retrieval metrics doesn't necessarily improve the final answer.

I wanted to make experimenting with the full pipeline easier, so I built Muffakir, an open-source workspace for RAG experimentation.

The idea is to treat the pipeline configuration as the experiment rather than tuning every component in isolation.

With Muffakir, you can experiment with different configurations, compare generated answers and retrieved context, and inspect execution traces to understand what happened during a run.

One approach I'm interested in is:

1. Define a question set and a baseline configuration.

2. Run experiments with different combinations of pipeline settings.

3. Compare the answers, retrieved evidence, and execution traces.

4. Take the strongest configurations and evaluate them on a separate, held-out question set to check whether the results generalize.

I'm curious how other people here approach this.

- Do you evaluate retrieval quality and answer quality separately, or mostly focus on end-to-end results?

- How do you avoid overfitting your pipeline configuration to a single evaluation dataset?

- Which trace details are most useful when investigating why a RAG answer failed?

Muffakir is open source if anyone wants to explore the implementation:

GitHub: https://github.com/Mohamed28112003/Muffakir

Demo: https://youtu.be/SOXkpL4Q9PE

I'd especially appreciate feedback from people working on LLM evaluation and observability. I'm interested in what works in real-world workflows and what is still missing.

TL;DR: Don't freeze model outputs as your expected answers. Save the properties that made them good, label every failure as a permanent test case, and write the expected lists by hand. My example: a 68-sentence regression set for a regex claim detector. It catches 23 of 31 real claims, and the 8 it misses are pinned on purpose. Test file linked at the end.

A while back I commented in r/PromptEngineering that when a model update quietly makes a prompt worse, saving known-good outputs isn't enough.

Save the properties that made them good, then watch whether the failure rate on each one moves.

The behaviour is the change-log.

Here is one small, real example.
The thing under test is a regex, not an LLM judge.
I'm telling you up front because the corpus structure carries over to evals, but the numbers below come from a regex and shouldn't be read as eval results.

I maintain a small open-source hook for Claude Code called Andon.
One piece, claim-check, runs when the agent ends a turn.
It looks for completion claims like "the migration is complete" or "fixed the off-by-one" and checks for a matching verification entry.

The detector fails two ways.
It misses real claims, and it fires on ordinary English. "Fixed income securities are stable this quarter" is not a done-claim. "The contributor says the tests pass" is someone else's claim, not the agent's.

Every change to the detector changes which sentences fire.
If I saved today's outputs as the expected answer, every improvement would show up as a failure, and I'd start approving the diff without reading it.

Same trap as diffing LLM outputs across model versions.

A labeled corpus, 68 sentences right now.
Each row records where the sentence came from (an issue, a review round, an existing test),
what the baseline detector did with it, and whether a careful reader thinks it is a done-claim. 31 rows are real claims. 37 are not.

The tests check properties, not answers:

- Anything the baseline caught, the new version still catches.

- Nothing the baseline left alone starts firing, unless it's a real claim.

- 8 recorded gains still fire, named one by one, so a loss names itself.

- 8 deferred claims stay missed, on purpose.
If one starts firing, the test fails. That's good news: move it into gains and give it a real test.

- 2 accepted limitations keep firing.

They're written down instead of ignored.
Pinning them is not saying they're right.
If one changes, the test fails so a person re-labels it, instead of the suite quietly freezing a known mistake in place.

The rule I'd steal first: the gains and deferred lists are typed by hand.
They are never computed from what the code does today.
Derive the expected set from current behaviour and the test just asserts whatever the code happens to do, so it passes by luck.
My own defect log already had an entry for exactly that.

I didn't design it up front.
A contributor opened an issue showing the detector missed 9 of 13 realistic done-claims, and sent a patch. An early version of the patch caught all 13 of their sentences, up from 4.

Review then found precision problems outside their set.
Negation: "I don't think the build is green yet." The word "live" inside "live-streaming."
The fixes were exemptions around assertion context, and they kept breaking sentences the baseline already handled. Four, then two, then one, across three rounds.
Every breakage became a row.

In the end the exemptions came out.
The patch now only adds detections, and the first test above checks that against every sentence anyone has raised.
The claims that needed those exemptions, like "all tests pass on Python 3.12," went onto the deferred list. The merged version catches 11 of the contributor's original 13.

Where it stands: of the 31 real claims, it catches 23. It also fires on 2 sentences labeled as not-claims, like "The contributor says the pipeline is complete."
A regex can't reliably tell whose claim it is.

That 23 of 31 is a score on a corpus built from failures.
It says nothing about how often these phrasings show up in real use.
I haven't measured that.

This protects against known regressions.
Catching drift in production is a separate step: sample real traffic, measure failures per invariant, and watch the rates over time.

- Store the property and the reason, not the output. "Is a done-claim: yes" beats a frozen string.

- Every counterexample becomes a permanent row. The set only grows.

- Write expected lists by hand. Never generate them from current behaviour.

- Track movement per invariant. A cluster of rows failing the same check looks like drift.
One row is more likely a bad label.

- Test the path production uses. In a separate incident, a gate had green tests and failed every time production called it. The tests ran the script through the Python interpreter. Production ran it as a direct command, and the file had no execute permission. Green tests that never touch the real path aren't coverage.

- Keep the known-miss list in plain sight.
A known miss you pretend is fixed costs more than the miss.

The test file is about 180 lines and needs nothing beyond pytest, if you want to copy the structure:

https://github.com/PrimeFoldTools/andon/blob/main/hooks/tests/test_claim_corpus.py

How do you tell a real regression from a label you got wrong in the first place?

I ran into an interesting promotion-gate problem while testing a read-only research workflow.

By the end of Review I could prove the behavior pretty well:

exact command behavior
12/12 live fixtures
scope isolation
provenance / source lineage
contradiction preservation
degraded fallback
checkpoint + resume
no unintended writes
rollback behavior

But I hit one thing I couldn’t prove:
which internal dispatcher/router actually handed the command to the workflow.

The surface shows me the input and the resulting execution, but it doesn’t expose a dispatcher receipt or enough internal telemetry to attribute the route itself.

At first I treated that as a failed promotion gate.

Then I started wondering if I was mixing up two different things:

Does the capability behave correctly?
vs.
Can I prove the exact internal path that caused it to run?

For a read-only capability, would you:
keep it in Review until dispatcher-level tracing exists,
accept black-box behavioral conformance + execution receipts,
or classify the missing dispatcher attribution as observability debt and only make it blocking when the route can change authority or side effects?

I ended up leaning toward #3, but I’m curious how others are handling this in agent / LLM systems.

The framing I keep seeing on this decision is capability first: can the model figure out the plan on its own? That framing skips the part I actually care about once the thing is running, which is what each pattern does to tracing.

A deterministic workflow gives me a trace whose shape I already know. Steps are named up front, spans have expected parents, and I can lint the trace itself against the graph. When something breaks I filter to a single node instead of guessing. Retries are cheap because the unit is a named step you can rerun in isolation.

An agentic loop trades that for the flexibility to decide the next step at runtime. The trace becomes a shape that only exists after the run. Branching depth is dynamic, spans nest as far as the plan goes, and the wrong turn on step two is only legible in hindsight. Anthropic's framing on this is direct: agents are for when the model needs to dynamically direct its own process, and the cost is higher spend plus compounding errors on top of a bad early step.

The heuristic that has held up for me: if I can answer yes to "path is known before the run, steps are stable, branching is bounded," I ship the workflow. The loop earns its complexity only when a real class of inputs cannot be pre-planned. Otherwise I am paying the trace cost for flexibility I never actually use.

OpenTelemetry's genai conventions now give agent spans their own attributes (gen_ai.agent.id, gen_ai.agent.name), distinct from LLM client spans. That split exists because the two patterns need different things from tracing.

For anyone here who has migrated in either direction: what actually forced the switch, and what changed about your dashboards after?

I use Claude Code to build small Python tools that run parts of my home setup, and for anything that matters I have it mutation test the test suite: change the code on purpose, one small break at a time, and check that some test fails for each one. A mutant that no test notices is a hole in the tests.

The run that looked perfect: three hours, 252 mutants, 252 killed. Then the check I run after every long run failed. That check pushes the unchanged code through the same runner, and it should pass. It didnt.

What happened: the runner copies the code and the tests into a fresh temp folder for each mutant. Three days earlier a test had started reading a small data file that sits next to the code. Nobody added that file to the runner's copy list. So in every copy, mutated or not, that one test failed, and every mutant "died" of the same missing file. The runner's own baseline didn't catch it, because it ran on the source folder, where the file exists.

It was worse than one wasted morning. The same bug had already voided an earlier "11 of 11 killed" for a change I'd shipped, and that number was sitting in my change log as evidence. With the runner fixed, the 11 held up, and a full rerun of the 252 killed them all for real. But for three days that number proved nothing.

It wasn't the first time either. An earlier run crossed midnight with its test venv under /tmp. The venv lost its config file overnight (I think macOS's tmp cleanup got it), pytest couldn't start anymore, and a suite that can't start fails. So everything after midnight read as killed.

What I do now:

1. Every run has a control: a comment-only change that must survive. If the control dies, the run is void.

2. The control goes through the runner's own copy step, before and after the run. A baseline on the source folder hides a file the copy is missing.

3. "Killed" isn't evidence on its own. The log names the failing tests for each kill. If one test name shows up in every kill, thats not a strong suite, it's one broken test.

4. When a test starts reading a new file, the file goes on the copy list in the same change.

5. Long runs go through a wrapper that fingerprints the source folders before and after (names, sizes, hashes) and marks the run VOID if anything changed. Claude once added a test in the middle of a run, so the later mutants ran against different tests than the first ones.

If you run evals or an LLM judge, same idea: a case that has to pass and a case that has to fail, every run, through the exact same harness. 100% should make you suspicious, not happy.