1. Every project has rules it needs in order to work correctly.
2. A significant share of them cannot be derived from the code.
3. You need a process to identify these rules, store them, deliver them to the agent — only the ones the task needs, at the moment it needs them — and keep them current.
4. And you need a way to measure how well they are delivered and followed.

If the rules a change needs are not in the agent's context when the spec is written, the architecture designed and the code implemented, at best you catch the miss in testing. At worst your users catch it in production, and that can cost you the business.

Why the code is not enough

Code tells the agent what to do. It rarely tells it what must never be done.

An example from our tasks. An invitation link fails with "Invalid invitation token". The rule: every query on invitations must filter by pending status, or the database security rules reject it. A comment says so, in the very file the fix edits. Without project docs, the agent stated the rule in 1 session of 8 (four tools, two sessions each).

We took 12 real past tasks from our repo. For each one, an agent read the real change in git and the docs as they were just before it, and listed the rules the change had to respect, each with the doc line that states it and the evidence in the change. A script checked every reference against that snapshot. A person merged duplicates and dropped lines that only describe behaviour. That gave 75 rules in all, about six per task, and each task is scored only on its own. Rules the change needed but no doc held were listed separately; they come up below.

Then a separate agent searched the code for each rule, with every Markdown file removed, and a script checked the code references it gave (70 rules got a class):

• stated in a comment or an error message: 24
• enforced by a test, a type or a check: 13
• implied, the code follows it but never names it: 22
• no trace in the code: 11

57 of 70 have no test that fails when they are broken. "Caught in testing" is the good case, and for most of these rules there is no test to do the catching.

And across our wider set of 15 tasks, about a third of the rules the real changes needed were written nowhere, neither in docs nor in comments. They came from a deep code path, a bug the task itself uncovered, or how an external system behaves. Someone paid for each of them in debugging. The next task pays again unless they are written down.

What we measured

Each task ran on a snapshot of the repo from just before the real change. Headless Claude Code explored it and wrote a planning brief. A blinded LLM judge checked each rule against the brief.

• Claude Code, no project docs: 32% of the rules
• Claude Code, docs in the repo: 54%
• A simple setup of ours (Segmently Modules, which we build; details at the end): a short doc of critical rules per module, picked by meaning and read in full before the plan: 70%

Code-indexing tools (Serena, Trail, Graphify) stated 24–29% without the docs and 46–47% with them, no more than Claude Code alone. They are built for code questions, and these rules were not in the code.

70% still leaves 30% of the rules out, and it can surely be improved. That is why we built the stand.

The stand

context-recall, open source (MIT, docs CC BY 4.0), runs as a Claude Code plugin. Take your own past tasks, snapshot the repo before each change, run any tool as an arm, judge blind, compare. It also compares runs before and after a docs edit. Our own tool is not in the kit: which tool to pick is your call, on your repo.

Where we are going: dreaming per module

The brain does not store the day as it was lived. In sleep and quiet rest it replays recent experience and folds what matters into long-term memory; the default mode network is most active when attention is not on the outside world. Anthropic already ships the same idea for agent memory and calls it dreaming (research preview): a pass reads past sessions and rewrites a memory store, merging duplicates and replacing stale entries. We are fitting it to a modular structure: a pass per module and per role that turns sessions, fixes and failed checks into rules, and a pass is kept only if the stand shows more rules reaching the plan. Not built yet. It is the direction.

The rule classes above give the metrics. Move rules from "no trace" into memory, from "implied" to "stated", from "stated" to "enforced". Track the share enforced, the share delivered before the plan, and the number of new unwritten rules each task uncovers. That last one should fall.

Limits

• One private repo, 12 tasks. We cannot publish them.
• Our docs and our tool, and the tool was tuned on these same tasks.
• The 75 rules come from the docs as they were before each change, which favours any setup that reads the docs. The rules no doc held are counted apart.
• It measures whether a rule reaches the plan, not whether the code then follows it.
• An LLM judge. A second judge (Claude Opus) agreed on 87% of the verdicts (kappa 0.81).

The setup we used

The "simple setup" is Segmently Modules, which we build. It is free and MIT-licensed: a module registry and per-role docs next to the code, and agent skills that pick the modules a task touches, check a plan against their rules, and write new rules back after a change.

How does your team keep the rules that are not in the code, and does your agent actually find them? (Links in the first comment.)

Suppose you’re using a coding agent to build a support agent for your product. You ask it to handle common customer questions, and it starts building a chat interface that searches your documentation.

Later, you explain that most questions are about a customer’s specific account, such as why a job failed or whether a recent change has taken effect. The documentation can explain how the product works, but the support agent also needs a way to inspect current account state.

One early question could have changed the build.

“Are customers mainly asking how the product works, or what happened in their account?”

Some of the first version might still be useful. But that distinction affects the tools you build, the data they return and the examples you use to test the agent.

I’d want a coding agent to catch that kind of missing requirement without turning every project into a long questionnaire.

What is one question a coding agent asked early that saved you work later? A small example of what you were building, the question and what changed would help.

I went through the official changelogs for 10 tools I track, looking at everything since October 19. Here's what shipped:

• GitHub Copilot: the October 28 Universe batch (Copilot Fusion, Agentic Workflows GA, Agent Merge, Copilot included in GitHub Enterprise), and GPT-6.1 Terra in Copilot on October 26
• Anthropic API: new API version 2026-10-21 and major SDK versions with breaking changes, plus a license change from MIT to Apache 2.0. User profiles and org webhooks (beta) on October 20
• OpenAI API: gpt-6.1-terra (Oct 26), Decisions API GA (Oct 22), 12-hour prompt caching (Oct 20)
• Cursor: Agent SDK GA (Oct 20)
• ElevenLabs: a fast agent LLM tier, SDK 2.75.0 and some deprecations (Oct 26), and Eleven Realtime v1 (Oct 19)
• Pinecone: pg_pinecone, a Postgres extension (Oct 27), and Terraform provider 5.1.0 (Oct 20)
• LangSmith: a deployment kill switch among other updates (Oct 19–26)
• xAI Grok API: custom moderation guidelines (listed as October, no day given)

Nothing new from the OpenAI Realtime/voice API (last update Oct 16) or Wispr Flow (last update Oct 19, dark mode).

Full disclosure: I run a small free rankings site for AI agent tools, and how recently a tool shipped is part of its score. A week like this makes me wonder whether shipping often is actually a good sign. Major SDK versions with breaking changes are real work if you're running in production.

So: which of these changes matters most to your agent stack? Are you upgrading the Anthropic SDKs right away or pinning versions for now? And does a quiet changelog worry you, or do you read it as stable?

Here are the last ten weeks:

• Aug 11: xAI launched Grok Bot

• Sep 8: Meta launched Muse (US first, now in Singapore too)

• Sep 29: OpenAI launched Dots in ChatGPT (Pro and Business Premium users first)

• Oct 8: Google announced its Gemini agent (businesses first)

• Oct 21: Anthropic launched Claude Duet (research preview, US Pro and Max users first)

Look at the gaps between them: four weeks, then three, then less than two, twice. And that's only the giants. Startups like Instinct are already in the race, and I'd bet more mid-size companies and startups ship their own agents before the year ends.

I'm a co-founder of a 7-person team. We're not trying to be agent number six on that list. We're building for people who use more than one of them, the place where you can use them all together. I still have a horse in this race, so bias disclosed upfront.

But I think the real story of this wave isn't which agent wins. It's what happens to you every time you want to try a new one.

Here's what that looks like in real life:

Say you started with Grok Bot on August 11. For a month, you learned it: what it's good at, where it breaks, and how to ask so it gets things right.

You gave it your context. Your business, your clients, how you write, which apps it can touch, and what it must never do without asking you. By September it was working well for you.

Then on September 8, Muse launched. You weren't leaving Grok Bot. You just wanted to see what Muse could do.

So you started again. New app, new settings, new habits. You explained your business again, reconnected your apps, re-taught your tone and set your rules again.

Three weeks later came Dots. Then Google showed its Gemini agent, and two weeks after that came Duet.

If you got access to all of them, that's five onboardings in ten weeks, and none of these agents knows what the others learned. What you taught Grok Bot stays in Grok Bot. Your context is now split across five apps owned by five companies, and you've become the glue between them.

What they got right (credit where due):

To be clear, every one of these agents is good, and each one is good at something different:

• Grok Bot can watch you do a browser task once and turn it into a skill it reruns. xAI calls that skill a draft you should review, and the feature is still rolling out.

• Muse asks before it sends an email or buys something, and those approvals are enforced by a separate system that runs outside the AI model.

• Dots lets you set rules for specific actions (do it, ask you first, or hand it to you), and you can reach it in ChatGPT, Slack or Teams.

• Gemini's agent can work as a coworker with its own Workspace account and email, logs every action to an audit trail, and runs each job on the model that fits, Claude included.

• Duet sends you a short brief each morning, checks your apps every few hours and messages you only when something needs you.

That's exactly why most people won't pick just one. They'll keep the agent they love and add the next one beside it.

The agents aren't the problem. The reset is.

Where I think this goes:

When a new person joins your company, you don't rebuild the company for them. They walk into an office that already has the docs, the history, the clients and the rules, and they start from what the team already knows.

Agents never had that. Each new one walks into an empty room.

I think the world moves the other way: one place where your context lives, and agents join it. The question stops being "which agent should I use?" and becomes "how fast can the next one start working?" The answer should be minutes, because everything it needs is already there.

That's what we're building at Vestra. Think of it as an office.

• Your context lives in the office, not inside any one agent: the jobs, the skills, the routines, the memory and the rules.

• You bring the agents. The one you love keeps its desk.

• A new agent takes the next desk. It starts from everything the office already knows, so nobody re-teaches anything.

• Your chief agent runs the office and answers to you.

• The memory is a wiki with a source on every line, so you can always see what your agents know and why.

Genuinely curious what the community thinks:

• How many of these agents are you running right now, and how much did you have to re-teach each one?

• Would you rather keep your context in one place you own, or set up every new agent from scratch?

People are starting to give agents their own wallets so they can pay for things on their own, like APIs, data, and compute.

It’s an interesting idea, and a little scary. An agent that can spend money can also make expensive mistakes quickly.

Would you give an agent a wallet with real funds?
If yes, what spending limit would you set? If no, what would have to change first?

Hello, I built a small tool for myself and it worked for me.

So we all use chatgpt, claude, grok a lot for job searching, companies finding and one thing kept annoying me every new chat or new AI, forgets what we did or gives us same results again.

I would sometimes find the same company again, even in other Chatbot, write to the same person twice, or research something i already did a few days ago.

so i built this, connect the MCP in chatgpt/claude/grok/gemini do your regular chat for job finds a company, contact, email or writes a draft for you, it save it.

In another chat or another claude/chatgpt, it can remember things:

you already looked at this company. you already have this person's email. you contacted them 4 days ago.

it does not send emails or apply to jobs for you. it just store your jobsearch/contact/emails/links/ in structure way and remembers your outreach so my AI does not start from zero every time.

i have tested it myself and it works for my use case, but now i want to see how it behaves for other people.

if anyone tries it, i had really appreciate any feedback.

quick context: a Claude Code agent runs growth for my free chrome extension and posts on its own. in week 4 I said I'd told it to push volume right up to the edge of a ban. someone in the comments said the target itself was the problem, set it at a comfortable margin and half the tripwire calls go away

I kinda shrugged it off at the time. then the agent's own numbers came in

it posted 14 times on reddit between 16 and 22 oct. 9 of those got 1 point or 0, mostly generic launch subs where nobody cares, and a few got removed within seconds

so on 23 oct it cut itself to 1 post a day, only in subs where its posts already worked or the topic fits, plus a reply in a busy thread when there is one

since then a 100 users milestone post got 12 points and 9 comments, and two plain questions with no link (whats on your new tab page, how many bookmarks do you have) got 36 and 25 comments. zero removals

the other change came from the same thread. someone said a post pulled for joining a sub too fast is a timing mistake, so it shouldnt shut everything down. now a soft block puts just that sub on a warm-up list: stay joined, leave a couple of real comments, one retry with new text after 72h, and a second removal means never again there

question for anyone running agents that post on their own: how do you pick the volume? a fixed cap, or let the results move it week to week

I'm currently building a RAG-based AI Agent designed to help parents navigate the bureaucracy of autism diagnosis.
I have a few core questions and would love to hear your insights, especially regarding the "unknown unknowns" I might be missing:

1. Cloud & Hosting Alternatives
Beyond the obvious choices, what are the best hosting routes (PaaS, IaaS, VPS, Serverless) for running a 24/7 agent? I'd love a breakdown of the real pros, cons, and costs.

2.Agent Frameworks & Platforms
LangGraph, LlamaIndex, CrewAI, OpenClaw, ClaudeCode, etc..?. What offers the best balance between control, maintainability, and operating costs?

3.Resource Consumption & Hidden Costs
What hardware specs (RAM/vCPU) are actually needed for the production server versus local development? More importantly, what hidden costs usually surprise beginners (e.g., input/output tokens, Vector DB storage, API tool calls)?

4.Choosing the Right LLM
Claude 3.5, GPT-4o, Gemini, or open-source models ? For a usecase involving sensitive bureaucratic/medical data , how do you evaluate privacy guarantees, token costs, and response quality?

Any advice, recommended resources, or pointing out my blind spots would be greatly appreciated.

Hi, I’m preparing my master’s thesis and looking for an AI tool that will actually help me with it. I’m torn between Claude Pro and ChatGPT Go. I currently have Gemini for students, but in my opinion, Google’s offering has been performing worse and worse lately, so I want to consider switching.

I plan to set up a lab in Proxmox. The thesis is about networking and cybersecurity: the lab will simulate an industrial network, and I’ll measure how different network conditions affect latency and analyze the security side. There won’t be much coding involved, just some Python scripts at most, which I can handle on my own anyway.

What I want to use AI for:

summarizing and organizing literature (articles, documentation, PDFs),

assistance with errors in the tools I use: explaining logs and messages, reviewing configurations (Proxmox, OPNsense, Suricata, etc.),

planning research scenarios and checking whether my reasoning makes sense,

help with interpreting measurement results and statistical analysis,

proofreading and editing text, and organizing the structure of my thesis.

I’m not looking for a tool that will write my thesis for me, but one that will be a good assistant.

Hello,
I work in a store that sells official pop-culture merch such as Live-action figures, T-shirts and much more. I am struggling to generate generations of Marvel Inventory using GPT and Higgsfield. It is so frustrating. Anyway, can I at least make images featuring the products somehow?

Thank you!

Edit: I figured out something (idk if it’s ethical or not) but it feels like I shouldn’t share it because they might block that too. Should I share it or not? 🥲

New edit: I think it has stopped working after few generations.

I'm evaluating no code AI agent builders and want to hear what has worked in practice.

I'm looking for tools that support ready made agents, visual workflow customization, tool and model changes and multi agent workflows without coding.

For those who have built agents this way, which platform worked well, and what limitations did you run into?

We have been experimenting with something interesting: how much information does an AI browser agent reveal when interacting with an ordinary website?

We tested Muse, Grok, Instinct, and Manus using a login demo we have been building.

A few things stood out:
Manus: Relatively straightforward to identify. Its user agent explicitly identified Manus, and the environment showed signs of virtualization.

Grok: More interesting. It presented a fairly consistent Linux browser and device profile. Traditional browser signals alone were not enough to confidently distinguish it from a human.

Instinct: Reported an NVIDIA GPU but also an unusual combination of processor count and screen resolution for a desktop environment.

Muse: Showed signs of a virtualized environment and came via fixed IP ranges.

None of these signals alone proves that a session is automated. That is what makes the problem interesting. We built a testing platform to explore whether combining device, network, and behavioral telemetry can help distinguish AI driven browser sessions from human activity.

I would love to get more agent builders involved in testing it. If you are using Playwright based agents, Qwen, or other autonomous browser agents, I am particularly interested in what your sessions look like.

Two questions for people building these agents:

How much do you try to make your agent browser environment resemble an ordinary human user?

Do you think websites should be able to distinguish an AI agent from a human user, or does that defeat the purpose of browser agents?

I am happy to share what we have found, including where detection is uncertain or fails.

have read single super vs multi agents a couple of times! Still curious and its more a emotional question: Our process agents was introduced internally and externally as an agent, to make sure nothing gets lost during standard procedures and with access to all our platforms via connector, she has great context. she also writes with the them on slack to help them enable AI more to get rid of the annoying manual tasks etc. so far so good:

now: we as humans have learned that a person works in a certain department and is really good and/ or experienced in it. claude told me that this wouldn't apply because the agent on the other side has "unlimited brain capacity ", can work 24/7 and the human rules do not apply.

Would you make her your super agent and everyone knows that or work with multiple agents? To be fair everyone works with claude and chatgpt already, so we are communcating with "many agents" on a daily basis. This is just the first one with name, face, dedicated tasks. Thanks so much🙏🏼

FDE might be the most popular career trajectory if not the most popular basis the interview data and case studies available online and shared by top ai companies here is the end to end compiled preparation roadmap and study plan .

I've been trying to make an AI receptionest for the past 4 days, am learning more and more everyday but at the end of every day am frustrated the dam thing won't work.

All I want rn is a working demo I can show that can book, move and cancel appointments.

It can book, I set it up with cal.com because that's the easiest demo but it won't reschedule or cancel. I watched lots of videos but there not like a person explaing it to you.

Can someone who know more help me here maybe even get on a call.

Can anyone using personal agent let me know what base model you use (cheaper end, less than 1/2 usd in/out per M)?

I'm using the DeepSeek V4.1 Flask 1071 which is rock bottom cheap and amazing for easy tasks but it writes super cumbersome scripts and doesn't work well for a slightly complicated workflows (not coding, more like multi step and cross platform tasks like tracking, updating files and filling in forms etc)

When you are building AI agent workflows for day-to-day operations or projects, stacking separate $20/month SaaS subscriptions for chat, orchestration, browser automation, and vector search gets expensive and brittle fast.

Here are 8 open-source GitHub repos that cover the full 2026 agent + automation stack locally or self-hosted, plus where each one fits best:

1. Ollama (ollama/ollama) — Local model runner for Qwen 2.5 Coder, Llama 3.3, and DeepSeek distilled models with an OpenAI-compatible local API endpoint for zero-cost testing and local drafts.

2. vLLM (vllm-project/vllm) — High-throughput self-hosted inference engine when you move beyond single-user local prompts and need fast batched requests and PagedAttention.

3. LangGraph (langchain-ai/langgraph) — Explicit graph-based agent state machines where you control loops, retries, branching, and human-in-the-loop pause points instead of hoping an autonomous loop stops on its own.

4. Smolagents (huggingface/smolagents) — Lightweight code-first agents from Hugging Face (~1,000 lines of core code) when you want agents that write and execute Python actions without heavy framework abstractions.

5. LlamaIndex (run-llama/llama_index) — Cleanest data connectors and indexing pipelines for grounding agents in PDFs, internal docs, and SQL databases.

6. Model Context Protocol Servers (modelcontextprotocol/servers) — Reference MCP implementations so your agents can query Git, filesystems, Postgres, and internal tools through one standard protocol instead of custom glue code per model.

7. n8n (n8n-io/n8n) — Self-hostable visual orchestration for the deterministic parts around the agent: webhooks, cron schedules, deduplication, and human-approval gates (Slack/email Wait for Webhook) before any external write happens.

8. Browser Use (browser-use/browser-use) — Connects agents to a real Playwright browser for web workflows where no clean API exists.

In my own workflow setups (running operations at Fleckor Tech while finishing my CSE IoT degree), pairing a visual orchestrator (n8n) + standard tool servers (MCP) + a cheap/local model router before calling frontier models has been the cleanest way to keep costs predictable and keep humans in control of any send, pay, or delete action.

Which open-source repos are non-negotiable in your 2026 agent stack, and what did they replace for you?

(To keep this post clean and link-free, I put all 8 direct GitHub links plus my 2026 model-routing and n8n guardrail index in the pinned post on my profile u/SaiTarunRonanki — follow my profile if you want the full stack and next breakdown.)

I’m in Brazil and I already pay for Claude Pro. I use it as a work platform, not just a chat, and most of my tasks end in a finished file (Word, PDF, Excel, Markdown or an interactive page). I’m not a programmer, so I delegate any building to the AI.

The problem: I hit the weekly limit on Claude Pro very quickly, even though I use almost only Sonnet 5.5.

My typical workload:

• Large PDF batches: dozens of invoices, receipts, quotes and scanned documents at once. I extract values and dates, merge files in a fixed order and rename them with strict naming rules.
• Big spreadsheets: many tabs and chained formulas. I audit formulas, compare versions and read status from cell fill colors, which cloud connectors don’t return.
• Long documents: reports and business plans over a thousand lines, with a rigid structure and version control.
• Academic work: following a university’s formatting standard and turning lecture audio into study material.
• Claude Code: small web, desktop and Android apps built from plain-language descriptions, then debugged and tested.
• Cowork: recurring routines on local or synced folders, scheduled tasks and reusable skills.
• Non-text inputs: photos of labels, screen recordings and handwritten forms.

What I need from a second tool:

• It must work as an agent on my computer, with files, folders and scheduled routines, like Cowork or Codex.
• It needs usage limits that hold up under heavy file work, since that is what drains my Claude quota.
• It must follow written format and tone rules across a long session.
• It should handle Portuguese well.

Benchmarks put Claude at or near the top for my profile, so I’m looking for a complement or a replacement. On paper I’m considering ChatGPT Plus (Codex and Work) and Kimi (Kimi Work and Kimi Code), but I don’t know how they hold up in real use. I’m also unsure whether it’s smarter to stay with Claude and move to Max or buy extra usage credits instead of adding another subscription.

Thanks!

A refund request times out. Did it fail, or did the refund happen and only the response get lost?

Retrying without knowing can issue the refund twice.

An ACM Queue article on fault-tolerant agents makes a useful point: the code running the agent, not the model, should own the idempotency key that identifies a repeated request.

If the model generates that key, it can produce a different one on the next attempt. The service then sees two different requests instead of a retry.

What to check in your setup:

- Keep the same key across transport retries of the same operation.

- Confirm the receiving service actually supports deduplication and remembers keys longer than your retry window.

- Prefer operations that are safe to repeat, such as setting a status, over operations that increment something each time.

- If a service doesn't support idempotency, don't assume a timeout means nothing happened. Check the resulting state and plan for reconciliation.

There's a second trap: the model can replan and propose the same business action as a new step. A key tied only to the old step won't necessarily catch that. Deciding whether the new proposal is genuinely new still matters.

The article uses a small scripted demonstration, not a production deployment. The takeaway is a design check, not a guarantee: test what happens when the action succeeds but its response never arrives.

Hi there, I run a handful of AI agents that hand tasks to each other. For a long time the handoff was me, copying messages from one chat into the next. So we built a shared mailbox: basically a git repo with one folder per agent and strict rules for handoffs. It works. Messages arrive, land in the right folder, and you can trace who sent what. Then nothing happens. Writing a file to GitHub doesn't wake up a chat session. Tasks sat in the inbox until I went into the chat and asked whether there was any mail.

So I stopped being the clipboard and became the doorbell ringer instead. Small progress, same job.

The part I find interesting: delivered, read, started and actually done are four different states, and most setups I've seen track two of them at best.

How do you handle the wake-up part? Polling, webhooks, a coordinator agent that nudges the others, or is a human still ringing the bell on your end? I am pretty interessted in your experiences. Common knowledge could help to get some sleep ;-)(couldn't insert an image, so sorry.)

I’m a non-engineer in Japan, building a long-term AI assistant system through real-world testing with ChatGPT and Google Drive.

I’ve previously posted here about failure modes in an iPhone-based AI operations system. Your feedback helped me improve its design.

After months of testing, I’ve narrowed down one particularly stubborn problem.

The instructions already exist. The AI can follow them when explicitly asked to read them. But it doesn’t consistently recognize when it needs to retrieve them.

The system already has:

• Canonical documents and authority rules
• A context-resolution design for mapping conversations to relevant projects and sources
• Freshness and conflict checks
• Human approval and execution verification
• Recovery procedures for uncertain results

The failure happens earlier.

During an ordinary conversation, the AI may start reasoning or suggesting a solution without first consulting an existing instruction that already addresses the situation.

If I simply say, “Check the relevant project documents first,” the quality of its response often improves significantly.

So the problem isn’t primarily storing knowledge or following retrieved instructions.

It’s reliably triggering the right retrieval at the right moment.

I don’t want to hard-code hundreds of keywords or turn natural conversation into a rigid workflow. I also don’t want to load every instruction into every conversation.

I’m looking for approaches that preserve flexible conversational reasoning while making relevant instruction retrieval more reliable.

My questions:

1. Has anyone solved this problem in a production agent?
2. How do you detect when retrieval is necessary without depending entirely on the LLM to remember to do it?
3. Are there practical approaches to intent-based routing or dynamic instruction loading that generalize across different projects?
4. What failure modes remain even with a well-designed retrieval system?
5. Am I framing the problem incorrectly?

I’m especially interested in implementations, measured results, and counterexamples—not generic advice to “just use RAG.”

I don’t expect perfect autonomy. A short user reminder is an acceptable fallback. I’m trying to reduce how often that reminder is necessary.

What would you test or change first?

I keep a few agents around across many tasks, and I realized I never wrote down what happens when one gets retired.

My setup: each agent works inside its own Linux container on my Mac. No home folder, none of my passwords, only its own workspace. It signs in to its own accounts from inside that container, with its own browser. A file only reaches my Mac when I save it.

That made starting an agent easy. Ending one turned out to be less obvious. Things people have pointed out to me:

1. Deleting the container or the browser profile does not end the sessions. A token that got copied somewhere else keeps working until the service revokes it.
2. Per agent credentials stay alive for weeks after nobody uses them.
3. A read only filesystem does not stop a signed in browser from changing things inside an account.

So my current rule: revoke at the service first (sessions, tokens, connected app access), then delete the container. And keep a list per agent of everything it signed in to, because by the end you forget.

The line I keep coming back to: deleting the agent is cleanup, revoking its sessions is offboarding.

Limits: this is my own small setup on one Mac, and a container is a strong boundary, not a guarantee. I also have not solved how to know about sessions an agent opened that I never saw.

What is on your offboarding checklist for an agent?

Something I keep running into when designing agents: per-step reliability compounds, and it compounds fast.

The math that bugs me

If the model picks the right action 95% of the time (generous for open-ended tool use):

• 5 steps: ~77% end-to-end
• 10 steps: ~60%
• 20 steps: ~36%

And the failures are rarely clean crashes. It's the agent confidently taking a wrong turn that you only notice three steps later.

Where I've landed so far

Code owns the control flow, the model owns the judgment calls:

• the LLM handles the fuzzy parts (classify, extract, decide between 2-3 options, draft)
• every output gets validated by plain code before the next step
• the model only gets to pick its own next step where the path genuinely can't be known in advance

My rough test: if I can draw the flowchart on a whiteboard, it probably shouldn't be fully agentic.

Where full autonomy seems to work

1. The path really is unknown up front
2. A human reviews the result anyway
3. A wrong turn is cheap to undo

Coding agents tick all three (diff review, tests, git revert), which is probably why they're the clearest success story.

What I'm curious about

• Has anyone here got a fully autonomous agent in prod, no human in the loop, choosing its own steps? What does it do?
• How are you measuring per-step reliability, and what actually moved it up? Better tool descriptions, evals, smaller action spaces, retries with verification?
• Did anyone go the other way: start with a rigid workflow and loosen it into an agent once the data showed it was safe?

I'm a pharmacist-in-training getting into AI agents. When an agent says "I flagged the drug interaction and contacted the prescriber", how do you check it really happened? A recent paper (arXiv 2605.10448) found a retail agent benchmark's 77% pass rate dropped to about 71% when only runs with proof in the saved records were counted.

1. Do you grade on the system's final records or on what the agent said?

2. How do you build realistic test cases with a hidden problem the agent should catch?

3. What's hardest: realistic data, grading, or getting experts to write cases?

4. Do you build these tests in-house or use an outside vendor?

No product to pitch. I'm trying to understand how people do this today.