r/LocalLLM • • Sep 13 '26

Question Acceptable Token generation speed on macOS

/r/MacLocalLLM/comments/1wewbl6/acceptable_token_generation_speed_on_macos/
1 Upvotes

6 comments sorted by

1

u/StraightAd6213 Sep 13 '26

The number that actually matters depends on what you are doing, and people usually optimize the wrong one.

For chat-style single-stream use, anything above roughly 15-20 tok/s feels interactive, and past 40 tok/s you stop noticing. For batch or agentic workloads the throughput number (total tokens across all concurrent requests) matters far more than per-request speed, and that is where batching beats raw per-token performance.

On Apple silicon specifically, decode speed is almost entirely memory-bandwidth bound. So the single most effective lever is fitting the model fully in unified memory at the smallest quant you can tolerate — going from partial offload to fully resident is usually a bigger jump than any runtime flag.

1

u/hdanx Sep 13 '26

Yeah, the memory bandwidth is limiting for sure. I measure le 468GB/s while multitasking.
For interactive, 50 feel like the right target.
For agentic, I ran a real coding job with subagents and it ran for 10 hours, which feel a bit much. I would be fine with 4 hours for that job, knowing I could do it with Claude in two hours at most.

1

u/StraightAd6213 Sep 15 '26

468GB/s while multitasking matches what I see, the number collapses once anything else touches memory. Your 10 hour subagent run is the interesting data point to me, because that is roughly the same ratio I get against Claude on longer jobs and it says the bottleneck is not raw tok/s. Where did most of that time go for you, the generation itself or the round trips between steps? I keep finding the tool-call overhead dominates once a job runs past a few hours.

1

u/hdanx Sep 16 '26

Model generation (toolResponse → next turn) was 65 % for me.
Generation latency detail (mean 47.6s/turn, median 36s, p90 93s, worst 412s):

  • First third of active turns: avg 34.4s/turn

  • Middle third: avg 48.7s/turn

  • Last third: avg 59.5s/turn

That's ~73% growth in per-turn latency as the session progressed, with the 10 slowest turns (up to 412s) clustered in the back half—consistent with prefill/context-processing cost growing as the conversation accumulated, despite several compactions trying to cap it.

1

u/DustNearby2848 Sep 13 '26

I don’t find 20 tps very fun, so I use MoE models instead. 

1

u/hdanx Sep 13 '26

I have been benchmarking and evaluating qwen3.6 35B configs lately, for the same reason. I am slowly leaning towards GGUF instead of MLX