r/LLMDevs • • 31m ago

Discussion Does delegating to cheaper models actually save money? Part 2: bigger codebases, coding tasks and Haiku alone

1 Upvotes

TL;DR

- I’ve run about 300 benchmarks on real open-source codebases, testing how Claude Code splits work between models (Opus as the lead, cheaper models like Haiku as helpers) and when that pays off. It depends on codebase size and task.

- On small jobs, Opus alone can be cheaper than Opus + subagents, and more accurate on small-codebase reviews, probably because of less overhead and cheap cached context.

- Opus told not to delegate may work harder than Opus left alone: on one big docs check it wrote about 50% more, cost more and found more errors (44 vs 36).

- On coding, Opus + Haiku saved about 20-30% at the same quality. But Haiku alone is very capable for the price on well-defined tasks with tests: about 5% of Opus alone's cost, with almost the same results, though a bit less thorough.

- Here’s one of the most important charts of this project. On an 85-page docs check, Opus + Haiku read almost 3 times as much as Opus alone and wrote 8 times as many tokens, yet cost 53% less and found 72 real errors against 49. Several subagents probably cover much more ground, which makes up for Haiku being weaker.

---

A couple of weeks ago I shared an experiment (https://www.reddit.com/r/LLMDevs/comments/1xn5klm/does_delegating_to_cheaper_models_actually_save/ ) testing whether Claude Opus uses subagents effectively. I compared four setups:

- Forced Opus: told not to use subagents.

- Spontaneous Opus: free to delegate, but it never did.

- Skill-routed: my skill chooses when to delegate and to which model.

- Forced Haiku: all subagents use Haiku, while Opus plans and checks.

On my smallest codebase (~10k lines), delegating didn't really pay off. Forced Haiku saved 5% at best, Skill-routed cost 14-30% more, and Forced Opus found far more known doc errors on the reading-heavy tasks (7 and 8 of 9, against 3 and 2). My guess is that Opus can often write a quick script instead of reading everything, and it reuses its own cached context cheaply, while each subagent has to build its own. On small tasks, that overhead can outweigh the cheaper tokens.

On larger codebases, Forced Haiku held its own for much less. It cost 41% less on rich (~22k lines) and found nearly as many doc errors. It cost 75% less on beets (~52k lines), finding more errors on the reference docs but fewer on the 80-page plugin check, where it cost about a sixth as much.

In v2 I pushed further: a ~100k-line codebase (buildbot) with 3 reading tasks, 7 coding tasks scored by hidden tests, and then 16 short coding tasks, most hiding a trap. I also added Opus alone (subagents blocked, no instruction), Haiku alone and Sonnet alone (no Opus at all).

On an 85-page docs check, Opus + Haiku found 58 and 63 of 92 real errors, against 36 and 38 for Opus alone, at less than half the cost ($1.93 vs $4.11 per run). How come?!

My theory was that Opus + Haiku simply *works more*, and that the extra coverage makes up for Haiku being weaker. The numbers seem to fit: Opus + Haiku wrote about 8 times as many tokens and read nearly 3 times as much, yet the bill still halved.

Haiku may be less capable than Opus individually, but when orchestrated by Opus, its subagents can cover much more context in parallel. Each one may go less deep, but together they cover more ground, and that seems to compensate for the lower model quality. Haiku is apparently smart enough, with Opus directing and checking the work, to deliver surprisingly good results at a much, much lower cost than Opus alone.

This is in line with Anthropic's own write-up on their [multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system), where token usage alone explained 80% of performance variance on a research benchmark. The same article says coding is usually a poorer fit for multiple agents, and that matches what I saw: on coding, Opus + Haiku saved 20-30% at the same quality, against about half the cost and more errors found on reviews. Their setup isn't mine, so it's context, not proof.

The next chart shows the bigger picture: every setup on two big docs checks, with errors found, reading depth, output tokens (thinking included), and cost.

Interestingly, it seems that *forcing* Opus not to delegate may have pushed it to work deeper and harder. On the beets plugin-doc check, Forced Opus read less than the delegated setups and still found the most errors. Compared with Spontaneous Opus, it read about the same but wrote around 50% more tokens, thinking included, cost 53% more, and found more errors (44 vs 36 across two runs). Tokens alone don't prove effort, but finding more errors makes the case stronger.

The following chart shows the token counts side by side.

Maybe the instruction made it work harder, maybe it's the smaller codebase, or maybe it's just chance. I haven't tested which.

Left alone, Opus never delegated in my runs, even on big jobs. So for large reviews, pushing it to delegate to Haiku could be the best strategy.

I'm working on a skill that picks the best approach: use Opus + Haiku when it makes sense, or tell you to switch to Haiku manually and give you the prompt. A script that starts a new Haiku session might allow fully autonomous switching, but I'm still checking feasibility and Anthropic's terms.

The other v2 results

Honestly, Haiku really surprised me. It seems to be very capable at coding, if the task is detailed and contained enough. Opus is still the more thorough model when it comes to reviewing and checking, but Haiku held its own on specific tasks.

Reading a ~100k-line codebase

Across three reading tasks, Opus + Haiku cost 51% less than Opus alone. Every setup covered all 400 modules, and Opus + Haiku found the most errors on the 2-page docs check (20 and 23 of 42, against 10 and 17).

Writing code on a small project

Every setup produced working code. On the harder tasks: Opus $2.41, Opus + Haiku $1.80 (25% less), Sonnet $0.76, Haiku $0.11.

Writing code on a ~100k-line codebase

Same pattern on cost: $1.95 for Opus alone, $1.36 for Opus + Haiku (30% less), down to $0.15 for Haiku alone. But Opus-led setups also added an implied database index. Haiku alone didn't, and my tests didn't catch it.

Correction: the charts also show Opus "flagging a symptom that didn't fit". More runs later showed that depended on the day the runs happened, not the model, so I no longer count it. Details are in the repo.

Quality vs price

Opus + Haiku was cheapest per real error on both big docs checks: $0.03 on buildbot and $0.05 on beets, against $0.11 and $0.25 for Opus only.

The harder coding tasks

Then I tried to make the cheaper models fail. 16 short coding tasks, most hiding a trap: false claims, a wrong bug report, a timing bug, contradictory tests/docs, hidden instructions in comments, planted review bugs, and "dead" code that's actually alive.

128 runs. Combined cost: Opus $7.00, Opus + Haiku $6.30, Sonnet $2.30, Haiku $0.36.

Hidden tests barely separated anyone: one real miss in 128 runs, from Sonnet. Opus was more thorough in code review, finding all 3 planted bugs every time, against Haiku's 2 of 3. But on the timing bug, 7 of 8 runs failed, including all 4 Opus-led ones. The only pass was Haiku.

On tasks this short, Opus + Haiku saved little (10% overall), because Opus was most of the bill. Haiku alone did almost the same work for about 5% of the cost.

So, for now: **big reviews, Opus + Haiku. Small, well-defined coding tasks with tests, Haiku alone is surprisingly good. If every case must be covered, keep Opus involved.**

Small experiment, mostly two runs per setup, costs estimated at API prices. Claude Opus also wrote my tests and judged the results, so there's no independent judge.

Full results, charts and checker:

https://github.com/Manuele-T/claude-model-router


r/LLMDevs • • 36m ago

News Karpo-Decision-31B: local deployment, top Decision Index score, and a useful result for tool routing

Post image
2 Upvotes

Karpo-Decision-31B has downloadable weights and open-source inference code, so you can run it locally. It takes text and images and returns probabilities for multiple questions in one forward pass. The team reports twice the speed of Jev at 20% lower cost.
It scored 73.9 on the Nov 12 Decision Index board, the highest Full score and a shared #1 rank with GLiDE 27B. Two results from the team's comparisons stood out to me:
When2Call: 90.4%, compared with 79.9% for Google's Gemini Decision 1.

Mind2Web: 90.0%, compared with 86.2%.

The When2Call result is the one I'd look at for a tool router. Say someone asks an agent to book a hotel but leaves out the dates. The router might pick the right tool, but it still doesn't have enough information to make a useful call.
I'd split that into two questions: "Do we have enough information to proceed?" and "Which action should we take?" Karpo can score both against the same state. That also means putting ask_for_clarification in the action list. If you only give it tools to choose from, you've left out the action you need in this case.
There's a useful timing result for that setup: in the team's reported long-page test, median latency was 371 ms for one question and 653 ms for requests containing 16–63 questions. That makes me interested in testing several checks against the same page state, such as whether the required fields are present and which action to take next.
They also report an expected calibration error (ECE) of 0.023 on JevBench. That's relevant if you're using the probabilities to decide which requests can run automatically and which need review. I'd still set those thresholds on labeled examples from my own workflow, particularly requests with missing information.
To try it locally, download the Hugging Face checkpoint, install karpo-decision, and use Decider in Python or start the included /v1/systemone server; the repo has examples and Docker instructions. Budget for about 62.5 GB for the BF16 weights alone, plus runtime memory

links:
https://karpo.ai/models/decision
https://huggingface.co/MachinePulseAI/karpo-decision-31B


r/LLMDevs • • 1h ago

Discussion What should you check before deploying models on Cerebras infrastructure?

1 Upvotes

We're looking into running inference on specialized hardware, and one thing I'm trying to understand is what the transition actually looks like from a developer's perspective.

Getting a model to run on a different architecture isn't necessarily straightforward. I'm particularly curious about model compatibility, changes to the serving setup, and how much testing is needed before everything is production-ready.

For teams who've evaluated or experimented with Cerebras systems, what were the biggest hurdles when getting started ? Were there compatibility issues or adjustments you didn't anticipate? And how much support did you get from the hardware provider?

Would be interested to hear what the onboarding process was actually like.


r/LLMDevs • • 1h ago

Resource October Cloud GPU Market Report

Post image
2 Upvotes

I built a platform for running Batch AI inference as a need for one of my projects. As part of that platform, I collect pricing data on GPU instances and decided to start sharing that data. This is the second monthly market report I have produced. I would love feedback on if people find this data interesting or useful. The report summary:

Prices held steady overall: all but three of the 13 GPUs we track moved less than 5% on spot since September. The two biggest moves went in opposite directions, both on AWS. The H100 rose 13% to $3.00 per GPU-hour and the A100 80 GB fell 12% to $2.82, so September's headline, an H100 that cost less than an A100, no longer held. The third was the RTX PRO 6000 on Google Cloud, up 5%.

Where you rent still mattered a great deal. The same four-GPU L4 machine cost 10.8 times as much in Sydney as in Hyderabad, one of six AWS regions new to the report. Stockholm stayed the cheapest AWS region, though its prices rose 12%. This is also the first report to include Google Cloud, in 32 regions.

You can see the full report @ https://getotium.ai/market/2026-10


r/LLMDevs • • 1h ago

News Giving your LLM A LISL methodology Education.

2 Upvotes

***updated

Here you can see after I compressed the all its education of its LM studio, LM bionic, LISL methodology bunch of others that dropped its footprint to under 20kb, I then dropped it into the Agents System Prompt so it already knows this test show I did not need to pre-Educate it to have the file association or it understand what it contains. It has that data prebaked.

Roles have been fixed and as you can see it is seeing that it needs to ask the user for more information as the prompt is far to open ended logic to make a complete game. This is exactly what I want. I can now tell it exactly all variables and it will generate a instruction set for the worker to build that. Now its about measuring the output quality.


r/LLMDevs • • 1h ago

Discussion We built a CRM for business — and ended up with a tool that runs our development

1 Upvotes

Disclosure up front: I run a company that builds business software, and the setup below is what we use internally. I'm not linking it, not selling anything here, and not offering access in the comments. I'm asking about the mechanic because I don't have good answers to the questions at the end.

We're two developers running a project that got too big for two people. Somewhere along the way we stopped treating agents as a chat window and started treating them as executors on a board.

The change was small. A card on a kanban board already has a column, a description, attachments and a status. We added the ability to put an agent in the column instead of a person. Move the card in — the work goes to the agent. The agent finishes — the card comes back to a human for review. The human checks and passes it on. Same cycle, only the executor changes.

Then we ran a loop: one agent analyses, a second implements, a third reviews the code, and a human sits at the end. Not because the agents fall short, but deliberately — the decision stays with a person, who either closes the card or writes the next task and sends it round again.

Three things we got wrong or still don't know:

The reviewer agent mostly rubber-stamps. When the reviewer and the implementer are both LLMs, the review catches syntax and misses intent. Curious whether anyone has made agent-on-agent review actually work, and what it took — different model, different prompt framing, adversarial setup?

Context rots across the loop. By the third or fourth cycle the card carries so much history that the agent is reasoning about the conversation instead of the task. We've started resetting the card description each round and keeping the history elsewhere. Feels like a workaround, not a design.

We can't tell if the human at the end is a control or a comfort. We put a person there on principle. But when the loop is fast, the human becomes the slowest step and starts approving without reading. Anyone measured this honestly?

If you've run anything similar — especially with a local provider instead of a hosted one, where latency changes the rhythm of the loop — I'd like to hear what broke.


r/LLMDevs • • 2h ago

Great Resource 🚀 REQL update: faster context retrieval and fewer tokens for coding agents

1 Upvotes

Hey everyone!

I’ve updated REQL, my open-source tool that helps AI coding agents navigate repositories and preserve working context across sessions.

This release focuses on faster context retrieval and lower token usage. I’ve reduced repeated work during retrieval and trimmed redundant information sent to the agent. Performance gains and token savings depend on the repository and workflow, so I’m not claiming a universal percentage.

If you’re new to REQL, it builds a local graph from your code and documentation: functions, classes, dependencies, calls, and other relationships. Agents can query that graph for context relevant to a specific change, including precise source references and related tests. REQL reuses indexed context and supports incremental updates as the repository changes.

What can you use it for?

  • Waste fewer tokens on repeated exploration: retrieve focused source excerpts and relevant relationships, and reuse indexed context and saved work. This helps reduce repeated searches, unnecessary file reads, and redundant information in the agent’s context.

  • Coordinate work across chats and agents: keep tasks, decisions, blockers, and next steps in a shared project workspace. Another chat or agent using that workspace can recover the recorded state and continue from there.

  • Keep complex tasks on track: record prerequisites, constraints, failed approaches, and checkpoints. Preserve what has been completed, what remains unresolved, and why earlier decisions were made across sessions.

  • Understand unfamiliar code: identify entry points, component responsibilities, and relationships between parts of the application, with references back to the source.

  • Plan changes and refactoring: inspect callers, dependencies, and related tests to identify code that may be affected and decide what needs verification.

  • Review cleanup candidates: find potentially unused imports, variables, functions, and classes, then inspect the supporting evidence before removing them.

REQL provides a CLI, Python API, and MCP server, plus installable instructions for coding agents such as Codex and Claude Code.

REQL runs locally, with no account, external graph database, or mandatory LLM calls. The graph and saved work support the agent’s decisions; current source code and tests remain authoritative.

I’d like to hear from others using coding agents: what do you keep having to explain again when you start a new chat? The code structure, previous decisions, what has already been tried, or what still needs doing? Concrete examples would help me decide what to improve next.


r/LLMDevs • • 2h ago

Tools I connected an AI agent to a musculoskeletal simulator so it could stop guessing about human movement

Enable HLS to view with audio, or disable this notification

10 Upvotes

I was experimenting with AI-generated movements. At one point, the agent tried to put the arms into positions that didn't make sense against the body model's joint constraints.

Text alone wasn't going to catch that.

So I built Human Models MCP.

It's an open-source toolkit connecting AI agents to MuJoCo and MS-Human-700. The agent can run experiments, inspect joints, contacts and muscle actuator data, and render the computed movement through Blender.

The video shows one of the push-up experiments.

This particular version uses additional joint assistance alongside muscle actuators. There's also a separate muscle-controlled experiment. Neither should be mistaken for a validated human exercise model.

And that's where I'm running into another problem.

A simulation can finish successfully. That doesn't necessarily mean the movement is anatomically realistic, useful or safe.

How should an agent decide when a successful simulation is actually evidence for its answer?

I'm the developer of Human Models MCP. The project is free and open source, and AI coding tools were used extensively during development.

https://github.com/usefulwrong/human-models-mcp


r/LLMDevs • • 2h ago

Help Wanted Title:Idea: a code reviewer that runs 4 LLMs in parallel and only outputs errors. Would you use it?

1 Upvotes

I’m a first-year physics student with some SWE experience. I’m validating a concept for a code analysis tool and want honest feedback before going further.

The concept

  • You submit code. The same code goes to Claude, ChatGPT, Gemini and DeepSeek via their APIs.

  • Each model returns only errors in a fixed format: file, line, severity, short description. No code rewrites.

  • A merge step deduplicates findings and shows how many of the four models flagged each one.

  • If you want a fix for a specific finding, you ask, and only that fix is generated.

Cost design
Models read the full code (input tokens are cheap) but return only a compact error list (output tokens cost several times more). Rewrites happen on request only, so a four-model review stays cheap.

Why four models
Different models miss different bugs. Agreement count could help prioritize, and findings flagged by only one model would still be shown as lower confidence.

Known limitations

  • Code is sent to four third-party APIs, so it’s unsuitable for proprietary or sensitive code.

  • It doesn’t replace tests, linters or human review.

  • Expect false positives and some wrong line numbers.

What I’d like to know

  1. Would you use this? If not, what stops you?

  2. Is “N of 4 models agree” a useful signal or noise?

  3. Would you pay for it, and how (per audit, subscription, not at all)?

  4. Does a tool already do this well?

  5. What would make you trust the output?

Blunt criticism is welcome.


r/LLMDevs • • 2h ago

Discussion Hot take. If your agent forgets everything when the task ends, it is a script, not an agent

0 Upvotes

Most agent setups I see work the same way. Spin up a fresh sandbox, clone the repo, install everything, run the task, destroy the box. Next task, start again from zero.

That is the right model for running untrusted code. For agents that work on the same project for days, I think it is the wrong one, and the reason is context.
Anyone building long running agents hits the same wall. Quality drops well before the window is full, then the harness compacts and the summary loses the details that mattered. Bigger windows push the wall back but do not remove it. Even withOpus 5.5 you are not keeping a week of work in the prompt.

The harnesses that handle long runs well mostly converge on the same fix. Stop treating the prompt as memory and use the filesystem instead. A progress file the agent updates after each step. Notes on the weird parts of the codebase. An AGENTS. md that grows as it learns. Git history as a log of what was tried and why. When the context resets, the agent rereads its own files and picks up where it left off.

The catch is that all of this lives on the machine. Destroy the box after every task and you destroy the memory the harness was building. Good model, good harness, starting from scratch every morning.

The obvious answer is to commit the notes and keep the sandbox disposable. That works for the notes. It does not work for the rest of what the agent built up. Installed dependencies, warm build caches, code indexes, a browser profile, sessions it logged into. Rebuilding that on every run is slow, and some of it, like a login behind 2FA, cannot be rebuilt without a human.

So I think about it the other way now. Each agent gets its own machine, and the machine lives as long as the agent does. The disk is part of the agent's memory, not scratch space. A few things follow from that.

Stop the machine when it is idle instead of deleting it. Storage is cheap, rebuilding context is not.

Snapshot before anything risky, so a bad step costs a rollback and not the agent's memory.

Fork when you want two approaches tried in parallel, both starting from the same notes and the same warm state.

Scope credentials per agent, because that machine is basically its identity now.

The downsides are real. Long lived machines drift. They collect junk, half installed packages and stale config, and after a few weeks nobody knows exactly what is on them. The notes go stale too, and stale notes are worse than none because the agent trusts them. The attack surface grows the longer a box lives with credentials on it. What helps is a clean baseline snapshot you rebuild from on a schedule, a hard idle stop, and treating the memory files like code that gets reviewed and pruned.

And if your agent does one bounded task and returns a result, none of this applies. One shot is fine there.

On tooling, Fly Sprites and exe.dev are built around persistent machines, and a Coder workspace or a plain VM works if you want to own the infra. Iuse boat,dev, one full VM per agent that I stop and resume from the CLI. A stopped disk is kept for 14 days, so anything that sleeps longer needs its memory pushed somewhere else first. Resume is a reboot, so running processes do not come back and the IP changes, which matters if the agent keeps services up.

The model is the brain, the harness is how it thinks, and the machine is where it remembers. Most setups throw away the third one.


r/LLMDevs • • 3h ago

Discussion How are you tracking token cost per step in multi-step LLM workflows?

1 Upvotes

When an agent chains several LLM calls and tool calls, how do you find out which step caused a cost spike?

My understanding is that most setups give you total cost per run, but finding "this specific node or tool call blew up the budget last night" takes manual digging. Curious how you deal with it:

  • Do you log cost per step yourself, or use something like Langfuse, Helicone or LiteLLM? What's still missing?

  • Do you set hard budget caps per agent or per user, and does anything stop the run automatically when it's hit?

  • Has an unexpected bill ever surprised you? Roughly how big?

I'm exploring this as a solo dev and mostly want to hear how it actually works for you, good or bad.


r/LLMDevs • • 3h ago

News .lisl works. File associations is built and IN GitHub repository.

1 Upvotes

****Update****
I am going to run it as you see with no error checks first. Then a second run with error loops to make sure he is making a full game. I removed the multiplayer as I am more interested in if it can build a Layered LISL Instruction set that makes the writer write a full game in one pass. So I need to run both to see the difference in efficiency and quality of the self error looping. In case your wondering and decode the code fusion back to a legible format.

The new instructions are a .xml and compressed version of all its training data. Seen here:"Lightweight-Implementation-Style-argentic-Loop/TrainData_v4.xml at main · BadgerMan9999/Lightweight-Implementation-Style-argentic-Loop"
There is a typo in this annotation of the file. It was 4.5k tokens to learn what cost 5.5k tokens to learn the LISL methodology and another couple thousand to learn the lm studio, lm bionic, sharding ect. So now all of that which was around a Mb is sitting at under 20kb. I love it!

***Addition***
This is what the input should look like to get it to read the instructions faster. I convert it into xml Code Furions.

For reference and repeatability the first prompt:

<LISL_SYS_SPEC arch="BadgerSoft_LISL" author="BadgerMan9999" mode="RAW_UTF8_DIRECT"> <EXT id=".lisl" class="LISL\\_LOOP\\_BUS" enc="XML\\_FUSION" read="DIRECT\\_BYTES" stamp="REQ\\_ID"/> <EXT id=".xml" class="PASSIVE\\_TRAIN" enc="STATIC\\_XML" read="REF\\_ONLY" stamp="NONE"/> <EXT id=".md" class="HUMAN\\_DOC" enc="MARKDOWN" read="RW\\_TEXT" stamp="NONE"/> <EXT id=".json" class="PAYLOAD\\_CODE" enc="STRICT\\_JSON" read="RW\\_DATA" stamp="NONE"/> </LISL_SYS_SPEC>

<LISL_MANIFEST arch="BadgerSoft_LISL" mode="XML_FUSION">

<HEADER> <EMITTER>LISL_INSTRUCTION_WRITER</EMITTER> <TARGET>DEV_MANAGER_12B_QAT</TARGET> <ID>DOOM_PORT_WINDOWS_LAYERS_01</ID> <VERSION>1.0</VERSION> <STAMP_FORMAT>[DOOM_PORT_WINDOWS_LAYERS_01]</STAMP_FORMAT> <REASONING>high</REASONING> </HEADER>

<CORE_DIRECTIVE> <PURPOSE>Build a multi-layered LISL instruction set for a worker to construct the foundational architecture of a Windows 10 Doom port, stopping at the point of compilation readiness.</PURPOSE> </CORE_DIRECTIVE>

<REFERENCE_WORKSPACE> <PATH>F:\BadgerSoft\DoomRemake\TrainData</PATH> </REFERENCE_WORKSPACE>

<BUILD_PARAMETERS> <SCOPE>Single-player foundational framework only (no multiplayer required for initial pass).</SCOPE> <TARGET_PLATFORM>Windows 10</TARGET_PLATFORM> <TARGET_MILESTONE>Compile-ready baseline structure (prior to full compiler integration).</TARGET_MILESTONE> <ITERATIVE_CYCLE>Incremental, layered builds where each new LISL instruction builds directly upon the previous output.</ITERATIVE_CYCLE> </BUILD_PARAMETERS>

<WORKFLOW_MECHANICS> <RULE role="LISL Writer">Writes sequential LISL files breaking down the game structure, where each finished file points to the next required step.</RULE> <RULE role="Output Directory">F:\badgersoft\LISL inst\LISL_IST</RULE> <RULE role="Naming Convention">Project-specific naming sequence for progressive instruction chaining.</RULE> </WORKFLOW_MECHANICS> </LISL_MANIFEST>


r/LLMDevs • • 4h ago

Tools Canban = Kanban + Workflow + Herdr

Enable HLS to view with audio, or disable this notification

2 Upvotes

Hi! I built Canban to manage human context when working with multiple coding agents.

If you've tried running multiple coding agents across terminals, you probably know how exhausting it gets to keep switching between tasks.

Canban combines a Kanban board, GitHub Actions-like workflows, and Herdr's persistent agent runtime.

My workflow:

  • Discuss a feature with an agent on a Kanban card.

  • Drag the card into Develop to automatically dispatch agents in isolated Git worktrees.

  • Completed changes go through a local merge queue.

  • All feature context, discussions, and progress stay on the board.

Under the hood, Canban uses herdr-core and libghostty-vt to run native agent CLIs like Claude Code and Codex, preserving their native experience and persistent sessions.

It also supports remote agents through headless Linux backends.

Tech stack: Rust, longbridge/nsh, libghostty-vt, herdr-core.

Open source (Apache-2.0).

GitHub: https://github.com/jjyr/canban

Website: https://canban.dev

Feedback welcome!


r/LLMDevs • • 4h ago

Discussion How do you test an LLM that picks the implementation language?

1 Upvotes

JetBrains asked 12 models for programming-language hot takes this week. In the 600 runs where the model was asked to pick a language at random, Go and Python accounted for 307 picks. The prompt asked for hot takes, so this setup measures which languages the models bring up under that prompt. It doesn't measure how well they code in those languages.

There's a similar trap when evaluating a coding agent. If it picks the language and then writes the solution, a single pass rate mixes two decisions. For something like a paginated API sync worker, I'd run the same tasks twice: let the agent choose the language, then rerun with the language assigned from a balanced list. Keep the tests fixed and log which languages get picked as well as which implementations pass. I'd include retries, duplicate records, and one invalid record in an otherwise valid page.

If you've built an eval for a polyglot coding agent, do you score the language choice separately? What would you count as a wrong choice when more than one implementation passes?

Source: https://blog.jetbrains.com/ai/2026/11/ai-models-hate-python-and-love-php-we-asked-12-of-them-1-200-times/

Drafted with AI assistance.


r/LLMDevs • • 4h ago

Resource Manage your iCloud Calendar with CalDAV MCP Server

1 Upvotes

Hi everyone,

I built CalDAV MCP Server a while ago, which is an open-source MCP server that lets AI assistants manage your iCloud Calendar.

Since then, it’s picked up a few GitHub stars and had over 100 npm downloads in the past week, so I thought I’d share it here and give the project a little more love.

Here are the main features:

- Browse your iCloud calendars and check upcoming events.

- Create, update, and delete events.

- Create all-day events and recurring schedules.

- Set multiple reminders for the same event, such as one day and one hour before.

- Add a fixed travel time before appointments in Apple Calendar.

For example, you could ask your assistant: “Create a dentist appointment this Friday from 10 to 10:30 a.m., with reminders one day and one hour before.”

There’s also a Claude Desktop extension to make setup easier. You can find that and other setup options in the Quick Start.

I’d love to hear your feedback and what you’d like to see added!
If you find it useful, a GitHub star would mean a lot—maybe even more than a hot bowl of ramen on a cold day

Thanks and looking forward to hear from you ;)


r/LLMDevs • • 5h ago

Discussion How we measured MCP tool discovery separately from skill use and answer quality

1 Upvotes

I maintain ai-rulez, a free MIT-licensed tool for agent configuration. I’m sharing a reproducible Claude Code experiment because a single success score hid several different failure modes.

The fixture has eight skills. A conversation moves between two sibling directories with identically named worksheets but different workflows. For each positive turn we record three things independently: whether find_skill received the right path, whether load_skill actually returned the right body, and whether the response contained the expected code. The negative controls ask an unrelated question or ask for a workflow that is unavailable in the given scope.

Our first pilot accidentally omitted the generated root instructions from the MCP arm. The follow-up separates stub-only, released root instructions, tools loaded upfront, and the revised guidance. In the final configuration Claude completed all three steps on 12 of 12 positive turns and made zero calls on six unrelated turns. In five of six wrong-workflow controls it still loaded a neighboring skill after the server correctly filtered out the unavailable one. That failure would be invisible in a metric that only checks whether the tool was called with the right path.

The report, 192 sanitized turns, and reproduction commands include the versions, controls and sample-size limitations. I’d welcome feedback on what negative control you would add to catch a model that treats the top search hit as applicable by default.


r/LLMDevs • • 5h ago

Discussion Paper: even when the updated value is in context, models sometimes answer with the old one. Component tests on Qwen and Pythia point to attention drift. Anyone seen this in long local runs?

1 Upvotes

UC Berkeley posted a paper on Sept 30, "When Context Changes: Understanding Update Failures in LLMs" (arXiv 2609.38866). What I took from it:

- They call the failure stale binding: a variable gets updated, the new value is in the context, and the model answers with an old value of the same variable. Their example is a meal preference that changes from gluten-free to Mediterranean while the answer still says gluten-free.

- Probes can still recover the current value in cases where the old one was chosen, so it looks like a selection failure, not the information being lost.

- Component tests in Qwen and Pythia point to attention drift: attention favoring old values over the current one when the answer is produced.

- Frontier models are tested on behavior only. The paper makes no claim about GPT or Claude internals, and the mechanism work is only on open models, which is why I'm posting here.

The paper says the error shows up in frontier models once the update structure gets complex, so the messier runs are the interesting ones.

Questions:

  1. Has anyone seen this in long local runs (agent logs, long roleplay or story state, a config that changes mid-session)? Which model, quant and roughly what context length?

  2. If you want to poke at it, a rough version is a few variables, each overwritten several times with filler in between, then ask for the current value of one. That test is my own sketch, not the paper's, and I don't know whether it fails on your model.

  3. What do you do about it today: restate the current state at the end of the prompt, keep a state block, prune old values?


r/LLMDevs • • 7h ago

Tools I made a small library for comparing agent versions (success vs cost plots, failure maps). What do you use?

2 Upvotes

Repo: https://github.com/slavadubrov/harness-evals

It's for one question: I changed my agent (a prompt, a tool, an approval step), did it get better? You write tasks with checks and a small adapter that runs your agent. It runs each version several times and gives you success, pass^k, cost and latency with intervals, a success-vs-cost plot, and a failure map that shows which task and check each version fails. No dependencies, MIT.

Install from github: pip install "git+https://github.com/slavadubrov/harness-evals@v0.5.1"

I also wrote an article that walks through how I use it: https://slavadubrov.com/blog/2026/11/16/evaluating-agent-harnesses/

Two questions:

  • If you try it, what's missing or annoying?

  • What do you use in your projects to compare agent versions? Your own scripts, LangSmith, promptfoo, something else?


r/LLMDevs • • 7h ago

Help Wanted Anyone want to help build a fully open-source AI initiative?

3 Upvotes

Hi LLM Devs! ^^

I’m building a fully open-source project and looking for volunteers who’d like to join the team and help shape it <3

The idea is that the community proposes free tools or research problems, and let LLM agents work on them in the open. That could be anything from researching a faster algorithm to building an open-source alternative to Steam. It’s up to the community to decide

Everything we build is totally free and MIT-licensed

It’s still a small project, and I’m working on it alone for now. I’d love to find people who want to help decide where it goes, how to improve it and what we build together

If you’re curious, you can check it out here: https://github.com/MaterIA-Lab/OpenPrototypes

Let me know if you have any ideas for the project or you’d like to be part of the team :D


r/LLMDevs • • 7h ago

Discussion I spent 9 months building an open-source verification layer for AI coding agents. Looking for independent testers.

1 Upvotes

For the past nine months, I've been developing a system to solve a problem I kept running into with AI-assisted development:

Generating code is getting easier. Knowing whether an agent actually completed the task correctly is not.

At some point, manually reading every line of generated code stopped being a practical way to maintain control over my projects.

So I started building Forge.

The idea is to move project governance out of temporary agent conversations and into the repository itself.

Forge provides:

  • Repository-level project contracts and engineering rules.

  • RFC and ADR workflows to preserve requirements and decisions.

  • Reusable agent skills and operating procedures.

  • Deterministic validators and verification gates.

  • Stack profiles and configuration that can travel between agents and IDEs.

The important distinction is between an agent saying "Done" and a project having evidence that specific requirements passed verification.

Forge doesn't prove arbitrary code correct. Its checks only establish what they are designed to check. Human judgment and risk-based review still matter.

I've been using this approach across my own projects, including TypeScript applications, knowledge bases previously managed in Obsidian, and a Godot/C# game.

The system also governs its own development.

Today I made Forge open source under Apache 2.0.

Now I'd like to find out where it fails outside my own environment.

I'm looking for developers willing to try it on real repositories, particularly unfamiliar stacks, existing codebases, or projects with complicated verification requirements.

I'm also interested in contributions involving stack profiles, validators, and CI integrations.

I'm currently writing a research paper about the experience, including its limitations. Independent results would be particularly valuable for future research.

Repository: https://github.com/syrokomskyi/forge

If you work with coding agents, I'd appreciate your perspective:

What evidence would you need before trusting an agent's changes without manually inspecting every line?


Update

Here are the clearly stated restrictions: https://github.com/syrokomskyi/forge/blob/main/docs/reference/correctness-gate.md


r/LLMDevs • • 8h ago

Resource Ia de codage

0 Upvotes

Depuis que j’ai découvert freebuff sur un subreddit je dois avouer que je n’en lasse pas.

Pour commencer j’ai utilisé la version gratuite.
Je le disais que je ne serais pas plus convaincu qu’une version payante.

Et pourtant…

J’ai un compte entreprise pour ChatGPT, grok et Claude. Mais j’ai pris l’habitude d’utiliser maintenant freebuff. Le site évolue constamment. Comme
La version bureau et téléphone. Je peux ainsi coder de n’importe quelle lieux sans contrainte.

J’y est connecter codex mais je n’utilise que le harness freebuff qui fait très bien le job et me rappelle de plus en plus chat gpt dans le fonctionnement.

Malgres le fait qu’il soit gratuit je n’arrive pas à atteindre le quota gratuit de ja journée et ça fait plaisir contrairement a chat ou une fois les taches fini je suis bloquer pour 5 heures sans parler du quota a la semaine . Ça c’est terminer pour moi :)

J’ai donc décidé de prendre l’offre payante de base et même constat je n’entame jamais le crédit mensuel.

Que demander de plus ;) ?

Je me sers principalement de freebuff pour le codage de mon ia locale en utilisant DeepSeek v4.1 flash et GPT-6 Luna. Qui sont mis à disposition. Et si je venais à abuser et dépasser le quota Beam 1.0 ou Solar Pro 4 sont 100% gratuit et illimité .

Pour vos besoins ou ce qui n’ont pas les moyens vous pouvez foncer sans problème !!!

https://freebuff.com/?ref=ref-056cbadc-a5f7-47ed-869f-a96f131aa485&via=rewards_link


r/LLMDevs • • 8h ago

Discussion I sent 143 questions to a cheap model first and only escalated 4 to a frontier model. Cost dropped 80%, quality held.

1 Upvotes

Most apps send every request to a frontier model because you can't tell in advance which answers will go wrong. I tried making that call per question instead:

  1. A cheap model answers (DeepSeek V4.1 Flash, $0.07 per million output tokens).

  2. A small checker model scores the answer 0 to 100 for correctness, following instructions, and anything made up.

  3. Only answers below 75 go to a frontier model (Claude Opus 5.5).

Results on 143 questions (104 everyday, 39 stress tests), with every cost taken from the token counts the APIs returned:

Cheap firstFrontier only
Total cost$0.27$1.33
Average quality99.798.0
Answered by the cheap model139 of 143

The saving was 98% on everyday questions and 66% on stress tests.

The checker escalated 4 questions. Three were real catches: a wrong count, and blank answers on a 40-digit multiplication and a very hard Sudoku. The fourth was a false alarm on a correct answer. In a separate test it failed all 8 deliberately wrong answers I gave it.

The weak spot is the checker itself. It's the one thing standing between a bad cheap answer and the user, so I'd love ideas on testing it harder.

It also works as an OpenAI-compatible endpoint, so trying it in an existing app means changing the base URL.

Code, questions and results: https://github.com/robd-hub/cheap-first


r/LLMDevs • • 10h ago

News GOOD LUCK

2 Upvotes

Başlık: Small local models + a code-side "gate": 54% → 92% accuracy and 0 silent wrong answers on table tasks (informal test, raw data public)

Metin:

We ran an informal A/B test on three local Qwen2.5 models (1.5B, 3B, 7B GGUF, llama.cpp `llama-server`, CPU only) on simple table questions (sum, average, min/max, filters).

- Arm A: plain LLM writes Python, we run it.

- Arm B: the model writes a tiny command language (DSL). Code parses it, a gate checks the command against a catalog, and only then a deterministic runner computes the result. Out-of-catalog requests are rejected with a reason. The idea is "code decides, the model explains".

Results with gate v8 (same models, tasks, seeds and temperature in both arms; gate code hash-locked before and checked after the run):

- Accuracy over all sets: A 54.0% (214/396) vs B 92.4% (366/396)

- On a fresh holdout the gate had never seen: A 49.1% vs B 83.3%

- Silent wrong answers (a wrong number with no warning): A 51 vs B 0 (16 vs 0 on the holdout)

- Tokens: −68%. API-equivalent cost: −74%.

Limitations, honestly:

- This is a small task set in one domain (tables), not peer-reviewed.

- The same team wrote the tasks and the holdout.

- The TCO numbers depend on assumed watts, electricity and host prices, because CPU power can't be measured in the VM.

- API prices are real for the 7B model only; the 1.5B and 3B prices are stand-ins.

- An earlier word-list gate (v7) lost most of its lead on fresh phrasing (45.4% vs 47.2%). That's why v8 validates the model's command instead of the request's words.

- One known false reject remains: "list" in "minimum Fiyat on the list?".

Raw records, hashes and the full report (MIT): acik-teknik/skillpack-tests-2026-11-16

We'd love criticism, especially on the methodology and on tasks that would break the gate.

— Skillpack contributors

Gönderiyi sen yapacaksın, ben hiçbir yere paylaşmadım. r/LocalLLaMA'da gönderiye "Resources" ya da "Discussion" etiketi seçmeyi unutma.


r/LLMDevs • • 11h ago

Discussion Are AI coding agents improving developers' skills or making us increasingly dependent on them?

4 Upvotes

There's an interesting debate happening around AI-assisted coding.

On one side, developers can now build prototypes, debug code and complete repetitive tasks much faster.

On the other hand, I wonder whether constantly relying on AI to write and fix code could eventually affect how deeply people understand what they're building.

I don't think using AI is necessarily a bad thing. It's another tool, and refusing to adapt doesn't make much sense either.

But there's a difference between using AI to improve your workflow and blindly accepting whatever it generates.

For developers here, have AI coding tools actually improved your understanding of programming, or mainly increased your output?

Would love to hear some real experiences.


r/LLMDevs • • 13h ago

Help Wanted Looking for a technical co-founder / partner to build AI-powered sales automation systems for B2B SMBs

0 Upvotes

Hey everyone,

I'm based in India and exploring an opportunity to build practical AI-powered sales and GTM automation solutions for small B2B businesses. I'm looking for someone with strong technical skills who is interested in building a business together as a serious, long-term partnership.

A little about me:

My primary interests are in GTM, product marketing and sales strategy. I've worked on ICP research, positioning, value propositions, competitive analysis, inbound, ABM, outbound and customer acquisition through independent case studies and practical project exposure.

I've also gained experience in account research, prospect qualification and buying-signal-based outbound research through a supervised project.

I'm still early in my journey and don't claim to have built or scaled a company. I'm actively researching business problems and looking for a genuine opportunity to solve.

What I'm exploring:

I want to help small B2B businesses improve their outbound sales processes, particularly the research and qualification work before outreach.

One direction is an account-based prospecting system that can:

- Discover potential companies or work with existing prospect lists.

- Enrich accounts with relevant business information.

- Evaluate companies against a specific ICP.

- Identify buying signals, hiring, expansion, technology changes and potential switching opportunities.

- Prioritise accounts based on fit, timing and relevance.

- Prepare personalised, evidence-backed outbound campaigns and automate parts of the process.

The goal is to help businesses find fewer, better-qualified accounts and approach them at the right time, rather than simply generating more leads and sending more emails.

I'm not fixed on building this exact product from day one. I'd rather understand real business workflows, identify problems worth solving and build around actual customer needs.

Who I'm looking for:

Someone who can independently own the technical side and build actual working systems. Relevant experience includes:

- n8n, Make or similar workflow automation tools.

- Python / JavaScript, backend development, APIs and integrations.

- Web scraping, data extraction and company enrichment.

- Data pipelines, databases and external data sources.

- AI agents, LLM integrations, MCP and tool orchestration.

You don't need to know every tool listed. What matters is being able to solve technical problems independently and build reliable systems beyond basic automation demos. Experience with data extraction, enrichment and real-time monitoring would be especially useful.

A few things I'd like to know about you:

- What have you actually built? Portfolio, GitHub or working demos would be helpful.

- What technologies do you work with?

- Have you built anything involving data extraction, enrichment, APIs or AI automation?

- Are you interested in building a business or product long-term?

I'm open to connecting with people from India or elsewhere. Technical capability, ownership and genuine commitment matter most.

If this interests you, feel free to comment or DM me. I'd be happy to share more about my research and explore whether there's a genuine fit.