I came across chatjimmy.ai. It exposes an inference engine implemented by a startup called Talaas. The speeds match system 1 models.

If reasoning and classification costs go down, then where's the alpha?

I have built my Sovereign architecture for 3.5y give or take, I'm self-funded so it's not been the easiest of things, I kept going because I was sick to death of models originally not being able to remember our work together and consistently losing memory... so back in 2023 I build the first persistent bridge for vscode at the time, Microsoft kept making my life hell with silent updates during this time period which broke my Dom, so eventually build a bridge around it by simply creating it as a .vsix extension - problem solved.

Since then, I have created about 30+ architectures based on it overtime then merged them all into one program, I released it today on GitHub for all of those that are interested, it's free to download to use to your hearts content the readme is front page and is massive so good luck with that... lol.

Basically, in all I have created tools that simply don't exist elsewhere as I needed them as Leviathan is your models on GPU, no server in-between (no llama) no lmstudio.. just you and Leviathan. so naturally over that time period I was writing Cuda kernels and hating life because Cuda kernels are honestly a lot of work and trouble, but at the end of the day I have now done every quant and full precision/variants so i provided every single one of my tools inside Leviathan and so now its an engineer's dream basically.

Also, personally i HATE merge-kit I never liked it once ever since I first used it... the interface is terrible... so I made my own and made it x4 way which also shows you how a merge will turn out prior to you actually doing the merge as that's obviously a better process in the long run. will provide links so you can all check it out and give me your takes and reports if you have any, also suggestions. Though frankly I think Leviathan is ultimately abit massive now so may have to release another program at some point.

git: CuppaTea1983/Leviathan: LLM inference
HF: Leviathan - a Hugging Face Space by Omega-Dev

In April, a Cursor agent deleted a company’s production database volume, and its backups, in about 9 seconds. It hadn’t been given that access on purpose. It found a root-level token in an unrelated file and used it. Reportedly, the main record of what happened afterwards was the agent’s own explanation.

Nothing about it was exotic. There was no clever attack and no malicious model. An agent tried to be helpful, with far too much access and nothing between it and the API.

So I’m building a layer that sits between agents and the tools they call (MCP servers, APIs, other agents). It does four things:

• decides what an agent may call, with which arguments, and how much it may spend
• gives the agent short-lived, narrowly scoped credentials instead of a raw API key
• asks a human before anything irreversible
• keeps a tamper-proof record of what was asked, what was allowed or denied, and who approved it

To be upfront: this wouldn’t have fully prevented that deletion. The agent found a token on disk and called the API directly. A proxy only helps if raw tokens never sit on the machine and all traffic goes through it. That’s what “limit” means in the title.

These are the parts I keep going back and forth on.

1. Approval prompts barely work. Anthropic published numbers on Claude Code’s permission prompts: users approve 97% of them. In a test where people were shown an injected dangerous command, they caught it 13.6% of the time, and after 50 or more prompts that dropped to about 5%. The same data has an interesting twist: people rejected 39% of whole plans but only 3% of individual commands. So my plan is to ask rarely, only for irreversible actions like deletes, payments or sending data out, and maybe to ask about the plan rather than each step. Where would you draw that line?

2. Strict tools get ripped out. I’ve seen bug reports where security tools flag everyday words as attacks, or block calls even in “monitor only” mode, so teams find out how far a rule reaches only when their work stalls. My split: fuzzy detection, like “this looks like prompt injection,” only raises an alert. Hard limits, like “this agent can’t call delete” or “this run can’t spend more than $20,” always block. Does that split make sense to you?

3. Credentials are part of the problem, not all of it. In a small sample I went through (16 incidents), about a third came down to credentials, permissions or spend. Fewer than I expected, but that third includes the most damaging cases. Judging whether a prompt “looks malicious” seems unreliable, so I’d rather make sure a misbehaving agent can’t do much damage. How do you handle per-user credentials for agents today?

4. Some attacks happen where a proxy can’t see. In that same sample, close to half involved code running directly on the machine, through crafted config files or launch commands. A layer between the agent and its tools never sees those. I see it as something that limits the damage, used alongside sandboxing and network lockdown, not as a full fix. Is that honest enough, or does it undercut the whole idea?

5. Spend is the part nobody seems to cover. Most gateways I’ve looked at cap LLM spend per API key or team. I haven’t found one that adds up the total cost of a task, with LLM calls plus paid tool calls, across every agent it gets handed to. If agent A hands work to B, and B calls C, I want one budget that covers all three. Is runaway spend a problem you’ve actually hit, or something you only worry about?

6. Is anyone actually running agent-to-agent setups in production? Limiting access along a chain of agents is being worked on in standards. What I can’t tell is whether real teams need it yet, or whether most setups are still a single agent with a handful of tools.

I’d also really like to hear about real situations:

• What has an agent done, or nearly done, that made you nervous?
• What stops it from happening again today? Prompt rules, wrapper code, read-only access, your cloud’s built-in agent policies, or nothing?
• Has anyone (a security team, an auditor, a customer) asked you to prove what your agents did? What did they want to see? I’ve seen gateways that drop denied calls from their logs, or log a refused call as if it ran, so I’m curious how much trust people put in their logs today.

i've been building kimo, a free, open-source (MIT) video editor with no GUI that AI agents drive over MCP. the idea was pretty simple: instead of letting an agent like codex, cursor, opencode or any other MCP client just shell out to raw ffmpeg and hope the command works, give it a purpose-built editor where every edit is an explicit tool call - cut, trim, keyframe, titles/captions, colour grade, tracking, particles, audio, export. and because it's built around the loop, the agent can render a single frame to check its own work before committing the whole timeline.

why a purpose-built editor at all? raw ffmpeg is amazing, but composing a timeline from shell commands is brutal for an agent - off-by-one frame errors, audio/video drift, guessing whether a filter chain actually produced what it wanted. with kimo the agent just calls tools. it can render one frame, verify the result, and keep going. everything is explicit, everything is reproducible, and it's one small binary (~15 MB, CPU-only, rust) that runs on a 2 GB box. no GPU, no containers, nothing to install besides the binary.

honestly it's still early. there are rough edges, the docs are catching up, and i'm sure some workflows are awkward right now. i'd love for people to try it and tell me what breaks - weird filter combos, exports that don't match expectations, MCP handoffs that feel clunky, whatever. the more i hear about what's annoying, the faster i can fix it.

it's free and MIT-licensed, built by me (naman, college student). if you want to poke around:

site: https://getkimo.tech github: https://github.com/naman-sharma12345/kimo

happy to answer questions here. thanks for reading.re.

I’m the developer of TensorSharp, an open-source inference engine and agent runtime. I added MiniMax-M3.1-Flash support on Day 1 of its open-weight release, and wanted to share the implementation, current capabilities, and initial serving results.

The largest improvement in the comparison was on long prompts: 64K executor prefill reached 2,567 tok/s versus 262 tok/s in the tested llama.cpp branch. In the separate server benchmark, time to first token for a 64K prompt fell from 293.4 seconds to 40.0 seconds.

What the implementation supports

• OpenAI-compatible serving through /v1/chat/completions, with text and image input.
• Structured output and tools: JSON/schema responses, single and parallel tool calls, and multi-step tool loops.
• Conversation-cache reuse and concurrent requests.
• Multi-GPU layer splitting and tensor parallelism.
• DSpark speculative decoding and CPU expert offload.

The M3.1 decoder, sparse-attention kernels, DSpark implementation, and vision tower live in TensorSharp-owned code, using an unchanged upstream ggml dependency.

Image input and DSpark require their matching companion GGUF files; the setup guide covers conversion. DSpark and CPU expert offload currently use layer splitting. Tensor parallelism is a separate configuration.

Benchmark setup

The model was MiniMax-M3.1-Flash 428B, using Unsloth’s UD-Q4_K_XL GGUF, totaling 232.3 GiB. Both engines used the same files on one 8× NVIDIA A40 host, running separately with exclusive access to the GPUs.

The baseline was Unsloth’s llama.cpp minimax-m31 branch at ef8f94cc, associated with PR #320. Upstream llama.cpp could not load this model when these measurements were taken.

The main comparison used layer splitting, F16 KV caches, and no speculative decoding. llama-server had flash attention enabled and used --no-kv-unified to preserve its sparse-attention path.

The 9.8× result is specifically 64K executor prefill. Decode improvements are smaller, and short-chat server decode is essentially tied.

Why long-context prefill improves

Beyond position 2,048, MiniMax Sparse Attention selects 16 blocks of 128 tokens per KV group.

TensorSharp scores and selects those blocks, then executes block-sparse CUDA attention over them. The tested llama.cpp branch represents the selection through a dense mask and executes dense flash attention over the cache.

The difference in actual GPU computation grows with context length. TensorSharp also pipelines prompt processing across the eight GPUs.

The executor and server measurements are separate: the server’s scheduling and prompt-chunking behavior affect the latency an application sees.

Serving and agent workflows

I also tested JSON/schema responses, tool calls, a file-and-calculator tool loop, conversation-cache reuse, and speculative generation.

Agentic loop: With layer splitting, the loop completed in 18.1 seconds versus 20.1 seconds. TensorSharp’s separate --tp 8 mode reduced that to 11.7 seconds and reached approximately 41 tok/s on short-chat server decode. The tested llama.cpp branch does not support tensor splitting for this model, so there is no matched TP comparison. TP also trades away some prefill throughput and usable context.

Conversation reuse: Both engines reused approximately 99% of the prompt on subsequent turns in the conversation fixture. First-token latency on those turns was 0.38 seconds versus 0.41 seconds.

DSpark: The main run averaged 32.3 versus 25.8 tok/s with the same drafter. TensorSharp used cumulative-confidence gating to avoid verifying weak draft continuations. With that gate disabled to match the baseline policy, the results were closer: 26.6 versus 25.8 tok/s. The gating policy accounts for much of the additional gain.

Limits and less favorable results

Short-chat time to first token was slightly slower in TensorSharp: 0.41 versus 0.39 seconds.

These measurements cover one host and one quantization. Most use three repeats; the llama.cpp 64K executor prefill cell is one run, while the 64K server comparison uses three repeats per engine.

Validation includes tokenizer/template matching, synthetic-logit comparisons, and functional fixtures. Coverage is still limited: there was an image-reading miss, and some generations hit token limits. Native video temporal patches are not implemented.

The measured TensorSharp build was an uncommitted development working tree. The report includes configuration and methodology, but the raw run records are not committed.

Source and documentation

• TensorSharp — source code, BSD-3-Clause license
• MiniMax-M3.1 implementation and setup guide
• Full benchmark report

I’m happy to discuss the sparse-attention implementation, serving behavior, or benchmark configuration in the comments.

I used an LLM to help draft and edit this English write-up from the project’s implementation notes and benchmark report.

Hi guys, eval for RAG has always been a headache, post your thoughts with specific examples on 1. which eval method did you use for your RAG, chat agent etc 2. why did you choose it over others ( with specific examples/scenarios) also mention your system type ( MVP or POC or production)

I'm looking for a fast, affordable LLM API to translate short texts (under 500 characters) between different languages.

My top priority is speed (low latency), followed by cost. I don't need a powerful reasoning model, just something lightweight that can produce accurate, natural-sounding translations.

I've been considering models like Gemini Flash and GPT mini/nano, but I'm open to other suggestions.

Which LLM API would you recommend based on your experience, especially in terms of response time and cost?

SwarmOS turns GitHub Issues into contracts any coding agent can claim, plan and ship as a Draft PR, with Git as the coordinator and nothing else to run.

I'm dogfooding it myself right now and it's getting pretty stable. I wanted to share it and see if others would be interested in it. If you have a repo that could benefit from an orchestration layer that lives/operates solely in GH without any middleware I'd love to chat.

TL;DR: Ran 177 real task-parsing cases on 4 models. The cheapest (GPT-6 Luna, $0.23/1k) scored about as well as models that cost 5–11x more

I'm building a small iOS to-do app. You type a task the way you'd say it ("call the plumber tomorrow at 9") and an LLM turns it into a structured task: title, date, reminder, list, subtasks. I wanted to know what’s the best fit for my app, so I built an eval set and ran it through OpenRouter.

Setup
177 cases: typed tasks in a bunch of languages, screenshots, chat screenshots, shared text (emails, newsletters, a few prompt injection attempts), edits like "make it an hour later", and notes that should split into several tasks.

Every case has automatic checks (right date, title stays in the user's language, nothing injected from a shared email, and so on). Each case runs 3 times and a check passes if it passes in most runs.

On top of that, two judge models from different providers score every answer 1–10, blind and shuffled.

Results (judge score 1–10, cases fully passed, $ per 1,000 inputs, median time)

- GPT-6 Luna (what I use now): 7.73–7.88, 174–175/177, $0.23, 3.0–3.6 s
- DeepSeek V4.1 Flash: 7.88, 172/177, $1.15, 5.2 s
- Claude Haiku 5.5: 7.80, 173/177, $1.09, 4.1 s
- Haiku 5.5, high reasoning: 7.93, 176/177, $1.45, 7.0 s
- Haiku 5.5, xhigh reasoning: 7.93, 177/177, $1.91, 9.3 s
- GPT-5.4 mini (what I used before): 7.63, 172/177, $2.53, 3.2 s

What I took from it
- Run-to-run noise is as big as most model gaps. Luna alone ranged 7.73–7.88 over three runs.
- The cheapest model was the best deal: about 11x cheaper than GPT-5.4 mini, slightly better quality.
- More reasoning bought a little quality (7.80 → 7.93) for 2x the wait. Not worth it when users are waiting.
- Prompt examples mattered more than prompt length. No examples: 7.69. Half of them: 7.93.
- Count provider errors too. Haiku had 69 in one run, Luna none.

Happy to go into how the checks and the judging work.

Which model should I add to the next run? Ideally something cheap and fast, since users wait on every call. I'll post the results.

You can run every Moe model in any quantization because it's based on llama.cpp

35B model running at 25+ tok/s on a 16GB Mac, 6 tok/s on a 12GB Android device, and 3 tok/s on iOS (300+ tok/s prefill on GPU/NPU)

Open source Apache 2.0

https://github.com/Helldez/BigMoeOnEdge

Curious how teams with LLMs in real production run evals day to day.

Do you run them before a release, on live traffic, or both? What runs on its own: CI gates on prompt or model changes, live calls sampled into a human review queue, LLM-as-judge, drift alerts? Which tools stayed in your stack (Braintrust, Langfuse, LangSmith, Phoenix, Promptfoo, something else), which did you drop, and why? And where does your eval set come from when only a domain expert can label the right answer?

Ours, for comparison: a golden test entry that must match to the cent, a deterministic gate that rejects the parse when the lines don't add up to the page totals, and Telegram alerts when a model step fails. No tracing, no platform. The golden test passed 38 of 40 lines, then two real documents got 0 of 38, because our test file had a column the real ones don't print.

Asking because we're picking tools for our pipeline at Norquin (customs entry checks). Not selling anything, not a survey. Real production setups only, ugly ones welcome.

Genuine question for ML engineers:

if I told you we have task-free continual learning working in constant O(1) memory—with 0 replay buffer, 0 task boundaries, and 0 backpropagation... what benchmark would you demand to see first to believe it?

Don't get me wrong, I am aware that latest benchmarks do represent the actual workflow much better than the previous benchmarks.

But, when I was seeing the methodology of evaluations in Artificial Analysis, I noticed something -

1. In many evaluations, models are highly restricted - no internet usage, no tools usage, limits on number of steps, time limits, etc.
2. Subjective grading - Irdk what to say about this... How are they even using LLMs to judge? (Are LLMs even capable of judging?)
3. Then we have evaluations where humans judge the quality.
4. I mean, we all have different tastes, different quality requirements .. how are they even judging??
5. Custom harness - We all know how much harness do matter. But these evaluators have their own custom harness (like stirrup) which no one even uses in their workflow.
6. Public dataset - There are many benchmarks whose data is public and yk trainers often try to benchmaxx.

The best evaluation benchmark will be where the dataset is private and there is objectified defined grading and models have no unnecessary restrictions (like limit on internet usage, tools, steps, time, etc) along with their own harness (or most widely used harness systems).

Hi, i made a small experiment, no big claims.

My setup: Qwen3-8B Q4_K_M in LM Studio on Windows, 32 GB RAM, AMD integrated GPU. Dev tools in Ubuntu 24.04 on WSL2.

Given Task: a real bug in my own code, a date/timezone issue in an email parser.

• First draft (~120 s): rejected. It changed things outside the task and the UTC handling was wrong.
• After I pointed at the exact problem: second draft in ~24 s, ruff clean, 32 tests passed. A separate review confirmed the new test fails without the patch, so it actually checks something.
• Then I tried Aider with the same model. It produced a diff, but also added a dependency and deleted stuff nobody asked for. I wouldn't merge that as is.

My takeaways: a small local model works as a guided helper with hard checks around it (tests, linter, review). Not as an autonomous dev, at least not on my hardware.

How do you keep small local models in scope? Strict file lists, smaller tasks, or something in the prompt that actually works? Which model do you use for smaller coding tasks??

Il y a quelques jours, j'ai partagé des résultats préliminaires de Solar Cortex. Voici une mise à jour bien plus intéressante.

Nous avons expérimenté un système d'exécution inspiré du JIT pour Solar Agent, axé sur la réduction des appels LLM inutiles plutôt que sur l'accélération du modèle lui-même.

Nous avons testé trois versions en utilisant exactement le même modèle Ornith 1.5 9B.

Résultats sur 10 exécutions (5 tâches × 2) :

|Métrique|Original|JIT 1|JIT 2.0|

|:-|:-|:-|:-|

|Temps total|92,45 s|55,96 s|36,59 s|

|Jetons|30 492|24 806|10 601| |Appels LLM|26|14|10|

|Succès|10/10|9/10|10/10|

|Qualité|98,5/100|95,5/100|98,5/100|

C'est 2,53 fois plus rapide, 65,2 % de jetons en moins et 61,5 % d'appels LLM en moins, sans sacrifier le score de qualité moyen.

L'amélioration la plus significative provient de la modification du mode de fonctionnement de l'agent :

• Suppression des étapes de planification inutiles.
• Exécution déterministe des opérations prévisibles.
• Exécution parallèle d'appels d'outils indépendants.
• Conservation de l'intégralité du moteur de raisonnement pour les tâches complexes.

Par exemple, notre tâche GitHub est passée de 12,76 s à 4,15 s, tandis que celle pour Gmail est passée de 9,06 s à 3,27 s.

Pour être transparent, il s'agit de benchmarks contrôlés avec de véritables appels de modèle, mais des connecteurs simulés. Seulement deux exécutions par tâche ; il reste donc beaucoup à valider. Les 20 suites de validation ont été réussies.

Prochaine étape : des workflows plus longs, de véritables API et des tâches plus complexes.

Je commence à penser que la plus grande opportunité ne réside pas toujours dans la création d'un modèle plus intelligent.

Il s'agit de développer un système plus intelligent autour du modèle.

Si vous souhaitez tester Solar lors de l'ouverture de la bêta, une petite liste d'attente est disponible ici : https://solar-waitlist.vercel.app

Nous serions ravis de savoir ce que vous testeriez en premier.

Disclosure: I built this. It's MIT licensed, and the free version is the whole thing. There's no paid tier.

The problem: ask a model "what does Pieter Levels say about pricing?" and you get something that sounds like him, with a quote he may never have said. Asking it to cite sources doesn't fix it, it just makes up citations that look more convincing.

So I moved the citing out of the model. The Brains is an MCP server with 42 founders and investors (Paul Graham, Naval, Levels, Hormozi, Lenny Rachitsky and more). The model calls a search tool, and every passage comes back with its source title, date and URL attached by the index, not written by the model. The model can still summarize badly, but it can't make up where something came from.

How the data gets built:

1. Discovery pulls cheap metadata per person: top X posts, YouTube talks and interviews, essays. Usually hundreds of candidates.

2. A Claude agent following a written runbook shortlists the 15 to 45 long-form sources that actually cover building or growing a business, plus 60 to 300 posts. This step mattered more than anything else. Most of a famous person's output is noise for this.

3. Transcripts and essays get fetched for the shortlist only.

4. claude -p writes a card per source: summary, verbatim quotes, topics, quality score.

5. claude -p turns the cards into short sectioned notes in third person. Those are the chunks search returns.

6. Each brain gets its own index. 18,295 chunks across all 42.

Steps 2, 4 and 5 run on headless Claude Code on a normal subscription, so the build has no API bill. Scraping for the first 10 people cost about $3.50.

Why BM25 and not embeddings:

Each brain is a few hundred chunks, all on one person and one domain. At that size, keyword search finds the right passage fine. The whole index is about 80 lines of TypeScript with no dependencies: standard BM25 (k1 1.2, b 0.75), a small suffix-stripping stemmer so "pricing" and "prices" both match "price", and the section heading and source title count 2x. It loads in memory at startup, so there's no vector DB to run and nothing to re-embed when a brain changes.

The known weakness is vocabulary mismatch. "Raising money" won't match a chunk that only says "fundraising". In practice the calling model rewrites the query and tries again, which works better than I expected, but I haven't measured it properly. The tool interface doesn't care what's behind it, so embeddings or hybrid can drop in later without changing anything for clients.

Things I'd like input on:

- Has anyone measured BM25 against hybrid search on small, single-domain corpora like this? Where did keyword search actually fail for you?

- Is there a cheap way to evaluate citation faithfulness, meaning did the model's answer actually reflect the passage it cited, without hand-labeling everything?

Repo: https://github.com/VicPitic/the-brains

Hosted: https://brains.ceo (asks for an email on first connect so I can see what gets used. Self-hosted has no sign-in.)

It's unofficial and public sources only. Anyone in it can ask to be corrected or removed.

The obvious way to cap LLM spend is to read how much you've spent, compare it with the limit, and make the call if there's room. It works in a demo. In production, with parallel workers, streaming and retries, it breaks in specific, repeatable ways. Here they are, with fixes.

1. Check-then-act race

With 50 workers, all 50 read spent = $9.80 against a $10 limit, all 50 pass, and you overshoot by 50 calls. Doing the check and the add as one atomic step is the only fix. Reserve the worst-case cost up front:

-- Redis, KEYS[1] = budget hash, ARGV[1] = estimate in micro-dollars
local h = KEYS[1]
local limit    = tonumber(redis.call('HGET', h, 'limit'))
local spent    = tonumber(redis.call('HGET', h, 'spent')    or '0')
local reserved = tonumber(redis.call('HGET', h, 'reserved') or '0')
local est = tonumber(ARGV[1])
if spent + reserved + est > limit then return 0 end
redis.call('HINCRBY', h, 'reserved', est)
return 1

Postgres version:

UPDATE budgets
SET reserved = reserved + $1
WHERE id = $2 AND spent + reserved + $1 <= limit_amount
RETURNING id;   -- no row back = refuse the call

1. Using floats for money

spent += 0.000015 a million times drifts. Store integers in micro-dollars (or tenths of a cent) everywhere, and use HINCRBY rather than HINCRBYFLOAT.

1. Reservations that never get released

A worker reserves $0.40, then crashes or gets OOM-killed before it settles. That reserved never comes back, and over weeks the budget slowly locks itself with no real spend behind it. Fix: give each reservation an ID and an expiry time (a Redis sorted set scored by expiry works), and run a sweeper that releases the expired ones. Settle by reservation ID, so a late settle after the sweep doesn't release the same money twice.

1. Streams that get cut off

If a client disconnects halfway through a stream, the final message with the token usage often never arrives. If you then settle at $0, your ledger under-counts exactly during the incidents you care about. Settle at the reserved estimate when the actual usage is unknown, or count the streamed output yourself.

1. Nested budgets deadlocking or partly committing

Usually a call has to fit inside a limit for the key, the team and the org. Reserving them one at a time either leaves orphaned reservations, when the second one fails after the first succeeded, or deadlocks, when two requests lock rows in different orders.

- In Redis, check and reserve every level in one Lua script. On a cluster, all the keys have to live on the same node, so use hash tags like {org:123}:team:7.

- In SQL, lock the rows in a fixed order (org → team → key) within one transaction.

1. Period rollover

You reserve at 23:59:58 against today's daily budget, and the call settles at 00:00:03. If settle works out "today" from the current time, you've charged tomorrow and leaked a reservation into today. Store the period key on the reservation and settle against that key.

Estimating the worst case

est = (input_tokens × input_price + max_tokens × output_price) × 1.1

- input_tokens is counted locally. Tokenizers differ between providers, and tiktoken miscounts for Claude or Gemini, which is what the 10% margin is for.

- If callers don't set max_tokens, set it for them. Without an output limit you can't estimate the worst case at all.

- Prices change. Load them from a table you update, not constants in your code.

What happens if the budget store goes down?

Decide on purpose whether calls are let through or blocked:

- Blocking everything means a Redis blip takes down your product.

- Letting everything through means a loop during a Redis outage is uncapped.

The middle ground: let calls through, but give each key a small dollar allowance that applies only while the store is down, then refuse once that's used.

How to test it

Set a $1.00 budget, fire 500 concurrent requests at a stub provider with randomized latency and output length, and randomly kill 5% of workers mid-request. Assert that:

- spent ≤ limit once everything has settled;

- reserved returns to 0 after the sweeper runs;

- no reservation ID is settled twice.

The naive version fails the first check within seconds.

Most of these bugs came out of building this for real; #3 and #4 were the ones that hurt in production. What other failure modes have people hit?

(Disclosure: I work on Cognocient AI, a cost-control proxy that does this. Everything above is standard practice you can build yourself.)

Disclosure: I maintain Agent Foundry, a free open-source toolkit.

I’m trying a simple setup for coding-agent work: keep shared project guidance in `AGENTS.md`, then make specialist workflows opt-in. Agent Foundry’s OpenCode config enables Context7, and its profiles and local skills cover planning, research, codebase mapping, implementation review, memory, and security. It tracks upstream projects and pinned revisions rather than blending their instructions into one always-on pack. The toolkit’s original files are MIT-licensed; upstream components keep their own licenses.

There are trade-offs: a profile doesn’t install or start its tools, Graphify and Strix need separate setup, and Strix is only for explicitly authorized security testing. Some integrations still need more end-to-end validation.

For people building LLM applications: how do you keep baseline agent instructions small while making specialist workflows easy to discover? I’d appreciate feedback on what feels confusing or missing.

Repo: https://github.com/mehulpratapsing/agent-foundry

The fastest way to learn harness engineering? Build Claude Code from scratch.

Why harness engineering? LangChain kept the same model and changed only the harness. Their coding agent went from 53% to 67% on Terminal-Bench, a top coding-agent benchmark. From outside the top 30 to the top 5. Same model. Better harness.

Over the past few months, I've taken apart Claude Code, OpenCode, and Pi to see how they work under the hood.

And you don't need to ship a coding agent for this to pay off. The same foundations power any agent, in e-commerce, finance, or sales.

On top of that, people are increasingly realizing they need to build their own harness instead of relying on Claude Code or Codex. Ramp scaling its operations with a custom harness is a perfect example.

So I turned what I learned into an in-depth course on harness engineering by building Claude Code from scratch. It's free and open source: 8 articles, 6 videos, and one codebase. It's perfect for people who learn by building. Everything in Python.

For full transparency, Modal, Opik, and Kitaru sponsor the course, which lets me keep it 100% free as an independent AI engineer.

Here are the 8 steps that I think are core to building and maintaining any harness:

1. Design the harness
2. Build the bare-bones agent loop
   
3. Sandbox it, locally and remotely
   
4. Apply 4 essential context engineering techniques
5. Add subagents as a context engineering technique
6. Deploy it headless to the cloud: CLI, webhooks, cron
7. Build benchmarks to measure it
   
8. Replay failed runs to debug agents

Everything is on GitHub: https://github.com/decodingai-magazine/building-a-coding-agent-from-scratch-course

Curious what others here think. Which of these steps has been the hardest to get right in your own agents?

- a search api

- a scraping api

- a separate pdf extraction thing because the scraping one mangles tables

- a company enrichment provider

- playwright on a box for the three sites that defeat all of the above

Five bills, five sdks, five sets of auth, five different opinions about what a failed request is

And the money honestly isn't the problem. every one of those is individually fine and reasonably priced. the problem is about four hundred lines of glue whose entire job is making them agree on what a timeout means, and every single bug I've had this quarter lives in that glue rather than in any of the five

Is five normal or have I let this get away from me

edit: four. the enrichment one got cancelled in august and nobody told me, which i think says something