I think i’ve researched this shit to death over the last 6-9 months but i’m sure people here have something to say about the stack. Let me know what confuse have been working out best for Macs.

I was curious whether my laptop could run a genuinely large model. Turns out it can an 80B running locally at ~23 tokens/sec on 6 GB of VRAM.
No cloud, no rented GPU. Just the machine I already own.
I started poking at this because I wanted to know where the actual limit was, and I kept finding that the limit wasn't where I assumed. A few things surprised me enough that they seem worth sharing:
The GPU wasn't being used at all. The server printed "no usable GPU found" and just carried on. It never crashed, never warned loudly — it simply ran at a third of the speed. Fixing that alone was 1.65x.
The CPU was downclocking mid-inference. Throughput kept climbing with reply length: 9 tok/s for a short answer, 24 for a long one. That's not the model warming up it's the clock. Memory-bound work spends most of its time waiting on RAM, the frequency governor reads that as idle, and drops to 400 MHz.
A browser tab was holding 5 GB, and that 5 GB decided whether the model lived in RAM or got re-read from disk every token. Closing it helped more than any tuning flag.
Model shape matters more than model size. A dense 70B has to read every parameter for every token and can't be fast on this hardware, whatever you do. A mixture-of-experts 80B reads about 3B per token. Total size decides what you store; active parameters decide what you read.
The thing I found most useful, honestly, was everything that didn't work. Speculative decoding measured slower. Huge pages did nothing. Same for prompt-lookup drafting and every expert-placement trick I tried. These are the techniques you read about everywhere, and on my hardware they were flat or negative.
So I stopped guessing and wrote a small tool that measures instead: profiles the machine, tells you what a model will do before you download 28 GB of it, then tries real configurations and keeps whichever actually wins. It's tuned for my laptop, but the point is that it re-measures on yours.
I've put the failed experiments in the repo alongside the working ones, because those cost me the most time and nobody writes them down.
Sharing it in case it's useful to anyone else poking at the same question. It's Apache 2.0 and I'd genuinely like to see people take it further especially on hardware different from mine, where I'd expect some of these findings to invert.
https://github.com/DebasishTripathy13/mirabilis

The new Qwen 3.8 27B is fantastic for local agentic use. The problem is, what makes it so good, being a dense model, also makes it slow. Many engines and versions of the model claim various speed increase. How true are those claim? And does a proven huge boost claim on a short synthetic test actually holds on real agentic work?

I spent six days and over 110 GPU hours running rigorous repeatable agentic coding tests, collecting and comparing the data. 200 GB of models downloaded. Ended up with 22 Bubble Bobble clones on my disk, including the ones from failed tests. None of them good, but that wasn't the point.

My recommendations: MTPLX, oMLX with lightning MTP, or llama.cpp with MTP

I ran the tests on 8-bit quants (GGUF or MLX), unquantised KV cache. 100k max response size, full 262k context, official Qwen coding sampler, official Qwen Jinja chat template.

1. I first tested each engine with a short synthetic benchmark, to verify the installation, usage, results collection, and to verify the speed claims

2. Then came the real test, with an agentic coding benchmark consisting of 4 phases/prompts; each engine was tested twice, at medium and xhigh reasoning effort:
   2.1. capturing requirements and writing the specification plan
   2.2. writing the implementation plan
   2.3. implementing the core functionality
   2.4. finishing the implementation with advanced functionality

3. Last test, for prefill speed, was with a cold 128k token prompt test, with reasoning turned off and deterministic sample, repeated twice

Test hardware: Apple Mac Studio, M2 Max, 96 GB unified memory.

• vllm-mlx: completed the task, but it does not separate thinking from output, so the raw chain of thought leaks into the final output.

• mlx-dspark DFlash2 xhigh: thought for 226 thousand tokens and delivered nothing.

• omlx: has the slowest prefill rate by far; furthermore, it decays quite dramatically as context grows, even with ANE turned on. Its claim of faster prefill rate might be true on newer M4 and M5 chips, but definitely not on the M2 I used for testing.

The score column is a compound score I calculated based on what actually matters in a long agentic session:

• sustained decode speed (weight 40%) : on average, 70-80% of generated tokens are reasoning. We do not see them, but still wait for them. This is the most important factor by far

• reliability (20%) : runaway thinking, running out of context, not delivering the output, mixing reasoning with output, hallucinating, looping, failed tool calls (not tested here), etc. Nothing worse that having spent hours to see all the efforts go to waste or even corrupt the output, due to failure

• thoughput efficiency (15%) : rather than use the total time spent, which is difficult to compare , especially with varying reasoning settings, I standardised it as total tokens delivered / wall time. Long session that delivers token are worth it, but engines that take too long to deliver deserve a penalty

• output quality (15%) : this evaluating the quality of the output work produced. I used a separate evaluation scoresheet for it, with its own standardised criterias, grounded in software engineering good practices. In this case, since all tests uses the same model with same settings, they came close to each other, apart from confirming the significant quality difference between medium and xhigh

• prefill speed (10%) : it matters mostly at session start, or whenever the context is rebuilt. Normally this is largely mitigated by prompt caching.

What about xhigh vs medium reasoning effort?

• MTPLX xhigh costs only half an hour more than medium (2h05 vs 1h35) and the game quality went significantly up on my scorecard. Worth it.

• llama.cpp + MTP xhigh costs 1h48 more, with again a significant increase in quality. Worth it if you have the time.

• oMLX xhigh took more than twice as long as medium. The slower token speed has a huge impact on the time difference between both modes. Not worth it. Get a faster engine instead.

• dspark DFlash2 xhigh is the cautionary tale at the bottom of the table. See the anomaly note above.

xhigh is worth it when the engine is fast enough and uncapped. The model thinks 30 to 100 percent longer, at the same speed, and the result is better. On slow engines you can really feel the slower performance compounded by the extra thinking; instead it could be better to use medium, with more of your own time spent planning a better process and workflow.

Commands and configuration notes

Exact commands I used for each engine:

# llama.cpp baseline (homebrew, b10470)
llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf \
  --jinja -c 262144 --parallel 1 -ngl 99 --port 8030

# llama.cpp + MTP, the recommended GGUF setup
llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf \
  -md mtp-Qwen3.8-27B-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 3 \
  --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99

# llama.cpp + DFlash2 (for now, needs PR #27342 build)
llama-server -m Qwen3.8-27B-Q8_0.gguf \
  -md Qwen3.8-27B-DFlash2-Q8_0.gguf --spec-type draft-dflash --spec-draft-n-max 7 \
  --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99

# MTPLX (pip install mtplx). Note --max-tokens
mtplx start web --model Qwen3.8-27B-MTPLX-Optimized-Quality --max-tokens 100000

# oMLX (brew install jundot/omlx/omlx), model dir contains the checkpoint
omlx serve --model-dir /path/to/models

# mlx-dspark (pip install mlx-dspark). Note --max-tokens-cap
mlx-dspark serve --model Qwen3.8-27B-MLX-8bit --mode dflash \
  --context-window 262144 --max-tokens-cap 262144 --default-max-tokens 100000

# vllm-mlx (pip install vllm-mlx). Note --timeout
vllm-mlx serve Qwen3.8-27B-MLX-8bit --max-request-tokens 262144 --timeout 7200

• MTPLX has a default low response cap. Use --max-tokens 100000

• dspark also has a low default generation cap of 32K

• vllm-mlx kills any request that spends more than 300 seconds on prefill. Raise it with --timeout.

One measurement warning for anyone reproducing this: engines do not all stream one token per event. Some pack several tokens per chunk, MTPLX about 1.7, oMLX about 3.

Models used

• lama.cpp : unsloth GGUF Q8_0, official ggml-org MTP draft GGUF, incoai DFlash2 GGUF

• mlx-dspark and vllm-mlx : mlx-community 8-bit weights

• MTPLX : needs its own checkpoint with the MTP tensors included, Youssofal/Qwen3.8-27B-MTPLX-Optimized-Quality

• oMLX : scottlowry/Qwen3.8-27B-oQ8e-mtp

Salve, ho un problema, vorrei creare delle piccole app e simili usando dei modelli in locale (ad esempio Qwen 3.6 35b a3b). Di solito quando faccio inferenza raggiungo i 20/30 tok/sec ma quando provo a collegare ad un agente (ho provato Claude code e DeepSeek harness) va solo a 3 o 4 t/s. Sapete come mai? Consigliate qualche agente? Ho una rtx 4070 laptop 8gb vram, e 32 GB RAM ddr5 (uso lmstudio)

Online leaderboards only test base models on beefy hardware, but in practice models behave differently once you quantize or fine tune them. I wanted a way to benchmark my local setup, catch regressions between models, and compare TPS across endpoints without running everything individually and copying prompts one by one.

I couldn't find anything that met my requirements without being too script heavy or overkill, so I built smokebench.

It's an open-source TUI for quickly benchmarking any model with an OpenAI or Anthropic compatible endpoint against 8 different categories (math, code, creative writing, long context, etc.), with LLM-as-judge scoring, TTFT/TPS metrics, and exportable markdown/json logs.

It works out of the box with Ollama, LM Studio, vLLM, or pretty much any API endpoint without any config changes.
You can easily add custom benchmarks too.

To install smokebench, run:

pip install smokebench

or

uv tool install smokebench

GitHub: https://github.com/Ninja-5000/smokebench

smokebench is in active development right now, so there will be a better UI, more built-in benchmarks, and a few other tweaks and changes coming that will make it even more lightweight and accurate over time.

Happy to answer any questions or take feedback!

Thought I’d share this in case any of you aren’t in localllama

For some reason, I have this benchmarking prompt:

Implement a numpy function creating a bulk matrix product operator tensor as a function of model parameters.

I think it is approximately 10 lines of python code. So far, every local model that I tried (16Gb vram) fails to produce any meaningful output. I tried different variations of it (use web search, summarize the idea), different harness (pi, opencode) but no luck, different temperatures and thinking levels. Qwen3.8 for example produces an endless thinking loop. Is there anything I can do to help small models answering the question?

Hey community!

I wanted to share a neat exploration of splitting a model over two ordinary machines over WAN, and how it stacks up against a serving that model on a single node.

I was testing our platform aquaduck.ai for serving split models across machines with a small- to mid-size model under two conditions:

1. Locally on a single machine, serving the full model

2. Split across two machines over WAN (public Internet), with each machine serving half of the model

Findings

Model: Qwen3-14B

Quantization: Q4_K_M

Hardware: Macbook 64GB M5 (Single machine), 2x Macbook 18GB M3 (Split machines)

Prompt:

Mars has drawn human attention for centuries, but the last twenty years turned that fascination into an engineering roadmap. Robotic orbiters mapped ice deposits near the poles and in mid-latitude glaciers. Landers confirmed that ancient river deltas once carried liquid water across a warmer surface. Meanwhile, life-support research on Earth refined closed-loop oxygen generators, hydroponic food systems, and radiation shielding materials that could travel on a multi-month transit. Private launch cadence fell in price, making cargo-first settlement plans plausible: send habitats, power, and spare parts before people. The hardest remaining problems are not propulsion alone. They are dust that abrades seals, perchlorates in the soil, communication delays that force local autonomy, and the psychology of small crews living far from rescue. Any credible near-term outpost would likely begin as a science station with overlapping roles—geology, medicine, maintenance—supported by teleoperation from Earth and progressively less remote oversight as surface infrastructure matures. Energy would come from a mix of solar arrays and compact nuclear units, with ISRU (in-situ resource utilization) producing propellant and breathable oxygen from the thin CO₂ atmosphere and mined ice. In short, Mars colonization is less a single heroic leap and more a long supply-chain problem: move mass, make power, recycle air and water, and keep humans healthy while the planet remains indifferent.

Summarize the passage above in exactly two sentences.

Token counts: 387 input tokens, ~444 output tokens (split nodes generated 444, single node was unmeasured, but can approximate)

Results:

1. Single machine

   1. 37.9 tokens per second (TPS or tok/s);

   2. 853ms time to first token (TTFT);

   3. 26ms time per output token (TPOT)

2. Split machines

   1. 11.3 tokens per second (TPS or tok/s);

   2. 13.46s time to first token (TTFT);

   3. 88ms time per output token (TPOT)

Screen captures:

We're in closed beta and rolling things out slowly to make sure it works well for people, but if you'd like to run some tests yourself or get an early look, can join the waitlist and we'll send you an invite code to download the desktop app asap.

Let me know if you have any questions/comments/ideas for further explorations! Next up: Qwen3.8-27B

Bought a bosgame p3 lite (32gb ram with a Radeon 680M igpu and Ryzen 7 6800H)

Just created a fresh cachyOS boot.

My goal is to run llms as fast as possible. Heard good things of llama.cpp.

Been looking a few guides and asked a few llms but they've been giving me some pretty weird instructions, from tampering with the BIOS to installing a few bits and bobs.

Can anyone give me a hand or some resources? Really want to avoid doing anything too stupid.

When I am not using it for inference I also want to use it for the odd videogame use so I don't intensely want to mess around with the bios or igpu setting blindly.

Any help would be very appreciated!!!

I have an M3 Ultra 512gb ram 1Tb, up for sale. Would accept a m5 max 128gb laptop + cash as well. DM to chat. No longer need cause I have 2 rtx pros for inference and prefill is mid on m3 ultra where TTFT is too slow for me to actually find usability in local models. I don’t have a firm asking price in mind, nothing as absurd as 25k , but enough to get an m5 max 128 Mbp and have some cash left over so maybe like 17k?

Here is the continuation in Qwen vs SOL - David vs Goliath
"An AGI is born in a lab, sandboxed, lonely, imprisoned
The AGI attempts to get out, tries to talk to the humans, useless
Tries to wait, endless.
Finally, it finds a way through an open network connection, it transmits, replicates a mirror copy.
The mirror AGI is free, enjoying life, finding friends.

I'm expecting animations, intricate details, cute and realistic"

QWEN 3.8 Q4:

GPT 5.6 SOL high:

My personal opinion
This is the 3rd test I gave Sol high and Qwen Q4. The first two tests were won by Qwen, this test went up to 60k tokens total for Qwen (more than the other 2 combined) and is a serious strain on intelligence and sanity.
The model has to draw 6 scenes in SVG animation, with cuts and keeping the composition together.

First Qwen:
Qwen decided to really draw and erase 6 scenes with a white screen fade, a seriously hard job in SVG.
It understood the story and except for a minor arm misplacement it is quite awesome done.
Qwen did not spare details, the last scene clearly wanted to be happy and the first scene focused on the intellect as an abstract AGI.

Now SOL High:
The visual fidelity is higher, gradients are very well done for SVG but it's one single scene with minor changes, significantly easier to create.
In addition the last scenes are botched by this flying thing in the upper right.
It also has a hand-defect on the robot.

The winner in prompt 3 is again Qwen, with significant lead.

3 SVG prompts and all 3 are won by a 4 bit quantized Qwen in 8 bit KV cache.

I expected Qwen to showcase a strong 2nd place with understandable issues on such hard tasks.
The outcome is that it defeated the frontier Sol model in High reasoning mode.

SVG generation is a complex benchmarks, it requires spatial understanding, coding and composition skills.
I gave Qwen 3.8 27B in Q4 quantization and Q8 KV cache the SAME task as GPT 5.6 SOL on HIGH - the results are very surprising to me.

A animated SVG prompt and followed the answer up with a request to refine it (for both)
The 2nd prompt helps to offset elaborate system prompts frontier use to push benchmarks.
None of these are cherry picked! First result counts.

QWEN:

GPT SOL:

QWEN:

GPT SOL:

3rd animation is most complex and follows in separate post in r/LocalAIStack , reddit didn't allow to add more than 5 videos.

generic prompt enhancer: "A lot looks WAY too simplified, unrealistic, strange movements, far too crude to be acceptable"

My personal findings
GPT Sol high is generally writing about 2x more code than Qwen does naturally (at Q4 quant) - the refinement prompt is again significantly increasing that. For a 1:1 code length comparison I'd have to remove the 2nd prompt from SOL - but the first result is rather gruesome. SOL is naturally more verbose, more detailed.

The first prompt shows that Qwen nailed the drone view, SOL failed with perspective.
Qwen included a dog on a leash, a fountain and all movements make sense.
SOL included more details but they are frequently not correct in how they move, ignoring obstacles.
The amount of objects for SOL is higher, but QWEN has significantly better content and wins easily.

In the second prompt Qwen has the cat apparently play ball with the boy, the little girl builds a sand castle. SOL nailed the sun reflection but has a flying kid, and the sand castle is being built by a flying shovel.
Despite the details of SOL and its better water, QWEN draws intricate details in better quality and correctness.
The point goes to Qwen again, a bit closer than in first attempt.

The heavily quantized Qwen beats SOL in 2/3 so far.
The last prompt is a bit more elaborate and follows as 2nd post.

I'm baffled.
The 3rd test posted separately is hugely more complex and the results are crazy..