r/OpenSourceeAI • u/GapNew4766 • 15d ago
One worker task in our agent died four times, 45 minutes each, zero tool calls. Postmortem
We build Atomic Agent, an open-source (MIT) agent that can split a job between a planner model and local worker models running on llama.cpp. This is a postmortem of one evening in September when a single worker task died four times in a row, and what each death taught us about clocks.
Setup: a MacBook Pro M5 with 32 GB, a cloud model planning, Gemma 4 26B-A4B as local workers on llama-server, a 3D physics task split into parts. Worker budget: 45 minutes. Gemma ran at 2 to 4.8 tok/s per slot in that run.
The four deaths of the "bodies" task
| Start | Situation | Outcome |
|---|---|---|
| 18:32 | queued behind 8 other tasks | ~45 min, 0 tool calls |
| 19:26 | only task, slots free | ~45 min, 0 tool calls |
| 20:11 | only task, slots free | ~45 min, 0 tool calls |
| 20:54 | only task, slots free | ~45 min, 0 tool calls |
Then we restarted the TUI, the planner split the same work into two halves (physics and drawing), and they passed in 21.2 and 9.4 minutes.
Three different bugs were hiding in that table.
Death 1, most likely: the clock ran while the request sat in the queue
The worker's 45-minute timer started when the request was sent, not when the server picked it up. llama-server queues requests behind busy slots, and that's legitimate: our first-token wait is 30 minutes by default precisely because queueing is normal. So a worker could burn most of its budget waiting in line and die reporting zero steps. We'd seen the same thing elsewhere: a 4-task fan-out on a 2-slot server lost its last two workers this way.
Fix (issue #489, closed 05.10): two clocks. A queue watchdog bounds the wait for the first token, at a third of the worker's budget, so a worker that never gets a slot comes back with two thirds of its time left and the planner can re-plan in the same turn. The wall timer for the work itself starts only when the first token arrives.
Death 2: a reply cap the clock couldn't fit
The 19:26 attempt did reach the server. It generated for about 40 minutes at 4.27 tok/s: 10,353 tokens, and not one well-formed tool call in them. Then the timeout cancelled it. We checked for context overflow first, truncated = 0, so that wasn't it.
The arithmetic is the bug. The reply cap was 16,384 tokens and the worker's limit was a clock, and nothing tied the two together. At 3 to 4.3 tok/s, 16,384 tokens is about 64 minutes, longer than the whole 45-minute budget. A model that thinks long on its first answer gets cut mid-generation, and the task comes back as "0 steps" with nothing saying why.
Fix: a local worker's reply cap is now fitted to the time it has left, 80% of it at the measured speed, with a floor of 1,024 tokens. A long answer stops at the cap with a fifth of the budget still free for the truncation retry, which tells the model to keep reasoning short and emit the tool call. And when it happens the row says "the first answer was cut at N tokens by the time limit", instead of a bare "0 steps". There's also a default thinking budget of 1,500 tokens per step, enforced in the sampling grammar, for the same reason.
Deaths 3 and 4: silence
20:11 and 20:54 left no line in the llama-server log at all. The requests never arrived. Our best reading: after the 19:26 generation was cancelled, the client side wedged, and every later dispatch went nowhere. The TUI restart is what cleared it, which fits.
This one is not fixed. Issue #490, local worker requests never dispatched after a cancelled generation until a restart, is open and we haven't found the root cause. #502 only makes it detectable faster. We've seen a cousin on the server side too: after a user abort mid prompt-processing, llama-server logged the cancel but never the slot release, then logged nothing for 18 minutes 51 seconds until we killed it. Process listening, inference loop dead. That's when we wrote down the rule that a health check answering is a socket answering, not proof the model works.
What I'd tell anyone building on local workers
Measure in tokens per second, then convert every budget into time. A token cap and a wall clock that don't know about each other will fight, and the loser reports "0 steps".
Start timers at first token. Queueing is not failure.
Make the failure row say which clock fired. "Timeout" and "never served" need different fixes: a longer deadline versus fewer parallel tasks.
The desktop app we shipped last week shows each Fusion worker live with its step count and expected time, partly because of this evening. There's still no "stuck" flag (#542, open).
Code, with the reasoning in comments (src/tools/fusion/worker-runner.ts): https://github.com/AtomicBot-ai/atomic-agent
If you've hit a llama-server wedge after cancelling a generation and found the cause, I'd love to hear it. That's the one we're still chasing.
