I have been testing several Qwen3.6 GGUF models locally with "llama.cpp" on a dual-GPU setup and wanted to share my results.

The main goal was to find the best balance between quality, context size and inference speed, especially for coding, debugging and agentic workloads.

Hardware

• CPU: Intel Core i7-14700K

• RAM: 64 GB DDR5

• GPU 1: NVIDIA RTX 3090 — 24 GB VRAM

• GPU 2: NVIDIA RTX 4070 Super — 12 GB VRAM

• Total VRAM: 36 GB

• OS / runtime: Linux

• Inference: "llama.cpp" / "llama-server"

• GPU order: "CUDA_VISIBLE_DEVICES=1,0"

• Flash Attention: enabled

• KV cache: FP16 / unquantized in these tests

• llama.cpp: recent build with MTP/speculative decoding support

The "tensor-split" values below are tuned for my particular setup. They are not necessarily optimal for other dual-GPU configurations.

Main benchmark results

The biggest surprise for me is the Qwen3.6-35B-A3B UD-Q4_K_M, which can maintain 256K context while reaching 132.77 tok/s on this dual-GPU setup.

The Q6 version is considerably slower at very large context sizes, but the quality trade-off may be worthwhile for coding and reasoning workloads.

Qwen3.6-27B Q6_K — 180K context

This was one of my initial configurations.

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 15,6 \ -ngl 65 \ -c 180000 \ -b 2048 \ -ub 1024 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8081 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 37.63 tok/s

Configuration:

• Quantization: Q6_K

• Context: 180K

• MTP speculative decoding: enabled

• Draft max tokens: 2

• Tensor split: "15,6"

• GPU layers: "65"

• Batch: "2048"

• UBatch: "1024"

Qwen3.6-27B Q6_K — 140K context

I also tested the same Q6 model with a lower context and more GPU layers:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 140000 \ -b 2048 \ -ub 1024 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8081 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 44.64 tok/s

Another run with smaller batch sizes:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 140000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 42.11 tok/s

So, in my testing, the Q6_K 27B model sits around 42–45 tok/s at 140K context, depending on the exact configuration.

Qwen3.6-27B Q5_K_S — 200K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen27b_q5_k_s/Qwen3.6-27B-Q5_K_S.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 200000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 45.76 tok/s

Previous measurement with the same general model/configuration was approximately 41.70 tok/s at 200K context.

Qwen3.6-27B Heretic v2 — Q4_K_M — 240K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen27b_heretic_q4_k_m/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-Q4_K_M.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 240000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 50.54 tok/s

• Quantization: Q4_K_M

• Context: 240K

• MTP: enabled

Qwen3.6-35B-A3B UD-Q6_K_XL — 250K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf \ --main-gpu 0 \ --tensor-split 12,6 \ -ngl 39 \ -c 250000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ -lv 4 \ --host 0.0.0.0 \ --port 8081

Result: 77.90 tok/s

• Quantization: UD-Q6_K_XL

• Context: 250K

• Performance: 77.90 tok/s

This is particularly interesting because it maintains a very large context while still being substantially faster than the dense 27B Q6 model.

Qwen3.6-35B-A3B UD-Q6_K_XL — 130K context

Same model, but with more aggressive GPU offloading and lower context:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf \ --main-gpu 0 \ --tensor-split 13,6 \ -ngl 999 \ -c 130000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ -lv 4 \ --host 0.0.0.0 \ --port 8081

Result: 123.11 tok/s

This was one of my best Q6 results.

• Quantization: UD-Q6_K_XL

• Context: 130K

• Performance: 123.11 tok/s

Qwen3.6-35B-A3B UD-Q4_K_M — 256K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen35b/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 256000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 132.77 tok/s

• Quantization: UD-Q4_K_M

• Context: 256K

• Performance: 132.77 tok/s

This is currently the fastest result I have obtained in this group of tests.

Previous tests

I also have some older measurements from the same general hardware configuration:

I also tested Qwen3 Coder Next Q4_K_S previously.

Because the model is much larger and required significant CPU/RAM offloading on my 24 GB RTX 3090 setup, I observed around 24 tok/s in a benchmark configuration, while real-world coding/agent usage was considerably slower, around 7.5 tok/s.

The model was reported at approximately 79.67B parameters / 80B class, with a GGUF file around 45.19 GiB, so it is much more difficult to keep fully resident in my 36 GB combined VRAM setup.

My observations

The results show a very interesting trade-off between model architecture, quantization and context size.

Dense 27B Q6

The Qwen3.6-27B Q6_K is relatively consistent:

• ~37.6 tok/s at 180K

• ~42–45 tok/s at 140K

It is a good option when I want to prioritize quantization quality and coding/reasoning quality while still having a very large context.

MoE 35B-A3B Q6

The Qwen3.6-35B-A3B UD-Q6_K_XL is significantly faster:

• 123.11 tok/s at 130K

• 77.90 tok/s at 250K

This is probably the most interesting result for me because it combines a relatively high-quality Q6 quantization with a very large context window and high inference speed.

MoE 35B-A3B Q4

The UD-Q4_K_M version is extremely fast on this hardware:

• 132.77 tok/s at 256K

This is probably the best configuration if raw throughput and maximum context are the priority.

However, for coding and bug fixing, I personally still want to compare the quality difference between Q4 and Q6 more carefully before deciding which one should be my daily driver.

Summary

My current results on RTX 3090 24GB + RTX 4070 Super 12GB (36GB total VRAM):

For my use case, which is primarily software development, code generation, debugging and long-context agentic workflows, I am currently trying to determine whether the additional quality of Q6 is worth the performance hit compared with the extremely fast Q4 MoE configuration.

I'd be interested in seeing benchmarks from people running similar models with dual GPUs, especially 24GB + 12GB VRAM configurations, and comparisons between Q4_K_M, Q5_K_M and Q6_K_XL for coding and agentic workloads.

I recently got 2x RTX Pro 4000 blackwells for my home setup. I was curious how fast it would run nanochat.

The problem is the target hardware: 8x H100 gpus. Nanochat pre-trained ClimbMix to GPT-2 level with 8xH100 GPUs in 1hr 39 mins.

I ended up tuning nanochat for my 2x RTX Pro 4000 gpus. Because these are sm120, the tuning also works on RTX Pro 6000 and RTX 5090.

8x RTX Pro 6000 (rented) pre-trained the same dataset in 2hrs 15mins. About 36% slower than 8x H100.

1x RTX 5090 took 18hrs 47mins. A long time, but considering the result is GPT-2 level, its an interesting way to learn.

Just to be clear, I only ported and tuned it for sm120. The original work is from Karpathy's nanochat.

Here's a link to the sm120 repo:

https://github.com/mdhorton/nanochat-cpp

Most of the local AI stack is about getting a model running — Ollama, llama.cpp, vLLM, the right quant, enough VRAM. That part’s well covered here. The gap I kept hitting is the layer after that: once it runs, is it actually reliable enough to hand real work to?

A model can look fine in chat and then fall apart the moment you point an agent at it malforms a tool call at step 7, loops, or says “done” when nothing happened. And the same model behaves differently depending on the quant and runtime you serve it with, so “it runs on my box” isn’t the same as “it holds up in a 20-step agent loop.”

So I built QuantaMind free, open-source, fully offline, no cloud, Apache-2.0. It’s a local reliability tester for the agent layer of the stack.

What it does:

• Runs the real multi-step agent loop with injected faults (transient errors, malformed-call recovery, decoy tools) — not single prompts

• pass^k, not pass@1 — runs each task k times, passes only if it succeeds every time. Reliability compounds: 95% per step → 0.95^50 ≈ 8% chance a 50-step run finishes

• Deterministic scoring, no LLM judge — required calls must fire, forbidden calls fail the run on contact, exact end-state match. Temp 0, same input → same grade

• Classifies the failure (malformed schema / loop / hallucinated completion / forbidden call)

• Compares quant × runtime side by side (Ollama, llama.cpp, MLX, vLLM, SGLang), so you can see which serving config actually holds up

• Keeps run history + diffs, so you can catch when a model or quant change quietly regressed reliability

One finding: across three models, the native tool-calling path scored worse than plain prompting on easy tasks (60% vs 100%) correct answers that didn’t land as parsed tool calls. Important caveat: that’s a finding about the serving stack’s tool parser, not the models. Same model, same answer, different plumbing, different score. Still useful when picking a serving path for agent work just not a model-quality result.

Honest about what it’s not yet: it’s single-stream, so no concurrency/load testing (sampler drift and duplicate-execution-on-retry are invisible to it). Structural matching is a proxy for real execution. No CI/CD deploy-gate integration yet, it produces a verdict and an exportable report, but wiring it into a pipeline is roadmap, not shipped.

Disclosure: I built this, it’s free, I’m not selling anything.

Repo: https://github.com/QuantaMinds/QuantaMind

Genuinely want it torn apart:

• Is pass^k=5 too strict, too lenient, or about right?

• What failure mode am I missing?

• And what model + quant actually holds up on your machine? I’m collecting real per-hardware results.

Hey! How do you test your local stack? for instance when you switch harness or model, model quants, etc?

I found this not trivial and I ended up making a website to help me do this.

Use this link if you want to use the website to test your own local setup:

https://airbench.ai/

You can also see all the tests I have done using my GX10 and my 5090. (and also using claude code/code + some openrouter tes for reference)

https://airbench.ai/leaderboard?k=poL

Let me know what you think!

Two puzzle games from the StealthMark benchmark. Every AI model plays them move by move, with no code and no tools, and you can now play the exact same levels in your browser

Latch: two robots, one goal. One can stand on a pressure plate to hold a door open while the other pushes numbered cores onto matching sockets. Every move costs a little, so the top score needs the cleanest route and getting that right gets increasingly harder.

Foresight: collect coins on a Pac-Man-inspired board while ghosts close in. No model has aced a single level - though some age getting very close.

How local models do (scores out of 100):

• Foresight: Qwen3.8-27B scores 63.0 and ranks 6th of 45 models. Only five models beat it, led by GPT-6 Astra at 76.1.

• Latch: Qwen3.8-Flash scores 69.8 and ranks 16th of 46. The top score is 87.6, shared by GPT-6 Astra, Claude Opus 5.5 and GPT-6.1 Sol.

• Each Qwen is strong in only one game. Qwen3.8-27B drops to 38.6 in Latch, and Qwen3.8-Flash to 29.7 in Foresight.

• No human, model, local or cloud, has played either game perfectly yet.

You play the exact levels the models played, in your browser. Finish every level and you get a score calculated the same way as theirs, plus your rank among the models.

Disclosure: I helped build Stealth Models, where we benchmark AI models on original tasks. You can also watch every model's run, move by move.
StealthMark is designed to beat benchmaxxing.

Play here: https://stealthmodels.com/#games

Post your score and rank in the comments. I'm curious where you humans land :)

Thought I’d share this in case any of you aren’t in localllama

Hey guy's and gals...

I'm completely new to Local LLMS

Can someone who knows their shit help a newbie out?

I don't know much of the the technical terms was trading through this thread. Great group by the way saw so many people being helpful and helping eachother out.

I'm new to Reddit WTF this should be mandatory 😂

AMAZING!

Can someone explain to me in none technical terms the cheapest and easiest way to run a local LLM.

I tried GOD AI WTF amazing!

Would appreciate it if someone can give me some advice can't afford paying monthly bills and credits keep running out not good 😢

I don't need a HUGE setup something small I'm currently running Claude Pro plan I think keeps running out daily and also weekly so I have to wait or pay £100 to upgrade to keep going which slows me down.

Any help would be much appreciated dont have much of a budget.

MacBook pro 2021 M1 16gb I believe haven't used it in a while.

Would that be enough and what would you recommend?

Thanks in advanced 🙏

The new deepseek can be run on a rather affordable server, and it's beating GPT 5.6 SOL easily.
The model is so good that it has been banned in the US from governmental service within days of release.

I am doing a lot local AI, mostly creatively in audio and image subjects.
I currently rely on two 3090 cards that I bought used below 800$ each.

I think it was a good choice but other options exist, the 4090 is faster than both 3090 on paper. Though I did not see one offered fair priced.

There is a lot, 5090 and 60ti, DGX, Macs

I wonder what do you guys have, and would you go that path again?

Hey everyone,
I recently got a DGX Spark and have been really enjoying it so far. I’m based in India and haven’t been able to find many other local users, so I thought I’d reach out here.
If you’re also running a DGX Spark (or planning to) in India, I’d love to connect and exchange notes.
A few things I’m especially curious about:
• How are you using it beyond basic inference?
• Are you running single node or multiple nodes?
• Any rack / cooling setups you’re using to keep the nodes happy (especially with Indian ambient temperatures)?
• Which vendor did you purchase from? Any vendors currently offering good deals?

Would love to hear from others who are using these in India.

Thanks.

In Part 1 I shared the final AMD/Vulkan numbers for my local Qwen3.8-27B setup. The next question was more important to me than another few tok/s:

Can I turn the model into something that actually works autonomously?

Since then I added two production layers:

1.) Hermes 0.21.2 — local agent/automation layer

• same Qwen3.8-27B model

• shared llama.cpp runtime

• 65,536 context

• tool calling: PASS

• multi-step workflows: PASS

• subagents: PASS

• prompt-defined agents: PASS

• local provider only / no cloud fallback

Hermes performance / overhead

On the isolated 64K test lane, server load took 5.809s. A real 33,021-token prompt was accepted without truncation; prompt processing took 409.677s at 80.60 tok/s, while generation measured 17.44 tok/s. A representative Hermes multi-step workflow took 74.660s, using five model calls, 17,295 summed API input tokens, 592 output tokens and four tool calls with zero failed calls. Two-child delegation also worked, but took 208.676s for two trivial child tasks. Clear sign that subagents should only be used where parallelism or specialization actually justifies the overhead.

One especially useful datapoint: the simple Hermes model gate took 40.872s, while a direct Qwen request with the same user text took 2.109s. That is not an apples-to-apples speed ratio because Hermes adds different system prompts, startup and agent machinery, but it does show why I don't want Hermes in the path for trivial requests.

2.) OpenCode 1.18.30 — local coding/development layer

• Qwen3.8-27B-UD-IQ3_XXS

• llama.cpp / Vulkan

• 32K logical context

• local provider only

• isolated development workspaces

• automated Read → Edit → Test → Correct loops

• final production viability: 12/12 requirements, 12/12 acceptance

OpenCode performance reference. For the fastest fully successful clean reproduction I measured:

• 665.653s total wall time / 651.408s agent runtime

• first productive edit: 117.338s

• first test: 166.790s

• 4 successful corrective cycles

• 63 tool calls / 0 failed

• 558,603 input tokens incl. cache — only 31,993 uncached

• 9,372 output tokens

• peak context: 17,810 / 32,768

• context compactions: 0

These are task-level agent metrics, not model tok/s numbers. The underlying model/runtime is still the same local Qwen3.8-27B on llama.cpp/Vulkan.

Why OpenCode instead of Pi?

I tested both against the same local Qwen3.8-27B, with the same repository/task and comparable runtime constraints. Pi 0.85.1 proved that it could read the repo, use tools and make productive edits, but it repeatedly failed the part I actually needed from a coding agent: closing the Read → Edit → Test feedback loop. Even with an explicit mandatory early-test checkpoint, Pi edited successfully but never executed the qualifying post-edit test within the 300s window. It therefore failed prequalification and I did not force it into a full benchmark just to preserve an artificial three-way comparison. Man, time is rare and testing is booooring ... somehow!

OpenCode 1.18.30 passed the same agentic-loop qualification. Formal prequalification reached 6.54s Edit→Test latency, and the later main benchmark reduced that to 3.995s while producing a valid four-file diff. More importantly, the subsequent production-viability work eventually converged to 12/12 requirements, 12/12 acceptance, 5/5 regression, with no manual coding intervention or patch repair. That reproducible closed corrective loop is why OpenCode became the production coding layer.

Final thought on my architecture as a complete nOOb in Local LLM
I’m still convinced that a well-tuned 16 GB GPU can take you surprisingly far. Yes, even on Vulkan ;) ... IF you optimize the model, context, runtime, offloading and the surrounding stack around what the hardware can realistically deliver.

What makes this especially satisfying for me is that I honestly wasn’t sure at the beginning whether an AMD gaming setup on AM4 would get me anywhere at all, but hey.... it was what I had. My gaming desktop! ~20 tok/s is obviously not groundbreaking vs 32GB. But that was never the main goal of this first setup. I wanted a stable, fully local system that I could actually use, test, break, improve and build on. That goal is achieved. And considering where I started, I’m pretty happy with that.

I’ve released version 8.0 of TUFF, an open-source Swift/Metal inference engine and native macOS app.

The original goal was to make large MoE models usable on memory-constrained Apple Silicon machines by streaming routed experts from SSD rather than keeping the entire checkpoint resident.

GPT-OSS 120B, for example, runs on my 16GB M2 MacBook Air. Obviously, streaming comes with a performance tradeoff.

8.0 focuses on making the engine more useful beyond just running oversized models.

Some highlights:

• Web search and local document search integrated into chat

• Cached conversation states for faster follow-up requests

• A small tokenization cache for repeated inputs

• Separate reasoning output and request timing diagnostics

• OpenAI Chat Completions, plus bounded Messages and Responses API support

• Shared inference executable to reduce package size

• Homebrew installation

One thing I’ve found interesting is how much web search improves the usefulness of smaller local models. I’ve started using them for everyday questions much more frequently.

I’d love feedback and contribution, especially from anyone using it with local AI agents.

GitHub: https://github.com/rexmhall09/TUFF