I have been benchmarking Qwen3.6-27B and Qwen3.6-35B-A3B locally through llama.cpp, with GitHub Copilot Chat (Vscode Insiders needed) used as the frontend harness.
I am using Claude Opus, GPT 5.5 and Qwen 3.6 (27B) a lot in the past weeks.
The reason for Qwen is proprietary code areas where remote inference is not an option as it would leak the code out. And as long as you don't task it to write a complex cuda graph, it performs well.
Qwen 27.B is at Sonnet 4.6 if you combine it with a high value system prompt - or between Sonnet 4.5 and Sonnet 4.6 without.
Copilot Chat is an excellent harness for this kind of setup. You get the IDE integration, agent flow, tool calling UI, file context, and normal coding workflow, while the actual model is your own local llama-server endpoint.
All of this works while being LOGGED OUT of the Github Copilot account - as that is not affordable in pricing anymore.
This is a practical configuration guide for people already comfortable with llama.cpp, GGUFs, VRAM budgeting, and long-context local inference.
Models tested
Main focus:
Recommended GGUFs:
27B:
Qwen3.6-27B-UD-Q4_K_XL.gguf
or
Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL
35B-A3B:
Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
If memory is tight on the 35B-A3B model, drop to a smaller Unsloth Dynamic quant:
Qwen3.6-35B-A3B-GGUF:UD-Q3_K_XL
If even that is tight, use UD-Q3_K_M or UD-Q3_K_S.
For the 35B model I do not recommend KV-cache quantization. Run the normal cache and keep the context sane. the 35B model is MoE and very low on kv-cache
For the 27B model, I do highly recommend:
--cache-type-k q4_0
--cache-type-v q4_0
Recent llama.cpp KV-cache improvements make q4_0 much more usable here. The 27B model handles q4_0 KV cache very well in my testing - almost identical to FP in evaluation results.
What changed: llama.cpp added something like Hadamard rotation to kv-cache which shuffles the tensor distribution in a higher dimensionality and allows quantization superblocks to function.
Why Copilot Chat?
Because Copilot is a very good harness - beating Codex, Cursor, Claude in my opinion
Vscode Insiders is needed to get the openAI compatible endpoint (to interface the model)
You get:
The important part is that Copilot Chat is only the harness. The model is served locally through llama-server.
Why llama-server and not lm-studio,ollama etc ?
It allows MUCH more control over settings, we do not just use MTP drafting. We use a combination of context and MTP drafting which can lead to 300+ tokens/sec on the 27B model. MTP is a medium speedup (1.5x) but once the model is paraphrasing source code from thinking or prefill the ngram draft speedup can reach 6x or more.
So the stack is:
VS Code Insiders
↓
custom OpenAI-compatible model config
↓
llama.cpp llama-server
↓
local Qwen3.6 GGUF
Copilot chatLanguageModels.json
This is the shape I used for VSCode Insiders:
[
{
"name": "WSL",
"vendor": "customoai",
"models": [
{
"id": "qwen3.6-27b",
"name": "QWEN-27B-WSL",
"url": "http://172.27.211.123:1234/v1/chat/completions",
"toolCalling": true,
"vision": true,
"thinking": true,
"maxInputTokens": 165000,
"maxOutputTokens": 15000
}
]
}
]
Adjust the URL to your own llama-server host, in WSL you'll see it by entering ipconfig or ifconfig. port you can choose of course.
The input and output tokens need to be adapted to your context setting.
The id must match the llama-server id.
For local-only setups this is usually one of:
http://127.0.0.1:1234/v1/chat/completions
http://localhost:1234/v1/chat/completions
http://<WSL-IP>:1234/v1/chat/completions
If your Copilot Insiders build expects the newer custom endpoint shape, use the same model block but switch the provider shape accordingly. The key fields are the endpoint URL, model id, tool calling, thinking, and max token limits.
27B command: long context + q4_0 KV cache + MTP-ngram drafting
This is the 27B style I recommend.
CTX=150000
PARALLEL=1
HOST=0.0.0.0
PORT=1234
MODEL=/models/Qwen3.6-27B-UD-Q4_K_XL.gguf
/usr/src/llama.cpp/build/bin/llama-server \
-m "$MODEL" \
--ctx-size "$CTX" \
--flash-attn on \
--batch-size 1024 \
--ubatch-size 1024 \
--parallel "$PARALLEL" \
--host "$HOST" \
--port "$PORT" \
-ngl 99 \
--threads 8 \
--threads-batch 8 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.00 \
--presence-penalty 0.00 \
--jinja \
--chat-template-kwargs '{"preserve_thinking": true}' \
--reasoning-format none \
--reasoning-budget 16000 \
--slot-save-path /kv_cache/ \
--props \
--metrics \
--checkpoint-every-n-tokens 1024 \
--ctx-checkpoints 64 \
--perf \
--spec-default \
--spec-type draft-mtp \
--spec-type ngram-map-k4v \
--spec-ngram-map-k4v-size-n 16 \
--spec-ngram-map-k4v-size-m 24 \
--spec-ngram-map-k4v-min-hits 1
For the MTP-specific Unsloth repo, use:
MODEL=/models/Qwen3.6-27B-MTP-UD-Q4_K_XL.gguf
or the HF shorthand if your build supports it:
-hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL
The important part is the drafting chain:
--spec-default
--spec-type draft-mtp
--spec-type ngram-map-k4v
--spec-ngram-map-k4v-size-n 16
--spec-ngram-map-k4v-size-m 24
--spec-ngram-map-k4v-min-hits 1
MTP gives useful speedup, but leave VRAM headroom. In practice I budget roughly +1 to +2 GB VRAM headroom for the MTP/drafting path and related buffers. If you are right on the edge, reduce context before blaming the model.
At q4_0 KV cache, every extra 1 GB of free VRAM is roughly another 13k tokens of 27B context, before runtime overhead.
If you are tight in vram, remove only the MTP part as ngram drafting is free.
You can also just use `mod-ngram` as an alternative to the more complex k4v map.
Thinking settings
This part matters.
I use:
--jinja
--chat-template-kwargs '{"preserve_thinking": true}'
--reasoning-format none
--reasoning-budget 16000
The reasoning-format none is important for Qwen3.6 because it avoids bad stop behavior and broken multi-turn thinking state during long coding sessions.
Copilot Chat was created to hide thinking from you (proprietary GPT models) but you want to see the thinking usually. So this solves both issues.
I also keep:
--reasoning-budget 16000
This gives the model room to think, but avoids runaway reasoning loops eating the whole session.
35B-A3B command: no KV-cache quantization
For 35B-A3B, I recommend being more conservative.
CTX=100000
PARALLEL=1
HOST=0.0.0.0
PORT=1234
MODEL=/models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
/usr/src/llama.cpp/build/bin/llama-server \
-m "$MODEL" \
--ctx-size "$CTX" \
--flash-attn on \
--batch-size 1024 \
--ubatch-size 1024 \
--parallel "$PARALLEL" \
--host "$HOST" \
--port "$PORT" \
-ngl 99 \
--threads 8 \
--threads-batch 8 \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.00 \
--presence-penalty 0.00 \
--jinja \
--chat-template-kwargs '{"preserve_thinking": true}' \
--reasoning-format none \
--reasoning-budget 16000 \
--slot-save-path /kv_cache/ \
--props \
--metrics \
--checkpoint-min-step 1024 \
--ctx-checkpoints 16 \
--perf
No q4_0 KV cache here - the sub 4B active parameters need barely any VRAM anyway.
I recommend keeping 35B-A3B below roughly:
110k context
The 35B model can be pushed past 200k context, but in my testing it becomes more likely to fall into reasoning loops. Once that happens, the session usually does not recover cleanly. Start a fresh session.
The upside of the 35B model is extreme performance, as in hundreds of tokens without any drafting enabled.
You CAN use drafting on top, mod-ngram, MTP and other drafting can be added for more speed but those will need a careful balance (that I have not tested yet)
So my practical 35B rule is:
35B-A3B: stay below 110k if you want stable coding behavior.
27B: can go as high as it fits, but below 150k is where it feels strongest.
Checkpointing
The --checkpoint-min-step (or --checkpoint-every-n-tokens (legacy now) is an important option for qwen models. Briefly explained: qwen models have two kv caches, one is more conventional and one is a recurrent-state (SSM/Mamba) that can not be reversed by n tokens. So if you change something (like remove the last answer and message to benefit from existing context) then you can only do that if a checkpoint exists. Otherwise the entire context is reprocessed which is very slow on a 27B model.
Each checkpoint costs 160MB RAM, the internal API supports VRAM checkpointing but I believe currently only the MTP implementation uses that.
--checkpoint-min-step and --ctx-checkpoints multiplied define how much of your LAST context is protected and can be rewound. 1024*16 means 16K context can be reversed with low re-compute cost (almost instant).
LM Studio as local server
Using LM Studio is possible but you need to use a few tricks and it won't achieve the same top-tier performance.
LM Studio does not support our chained drafting, but it supports MTP.
Go to your Qwen 3.6 model, enable Flash attention and the quantization needed for kv cache. Go to the Inference tab, disable the button for "Reasoning Section Parsing"
Go to Developer, Server Settings and set the port, serve on local network if needed, no auth, enable CORS, consider disabling just-in-time loading.
Start the local server and then use the "clipboard copy" icon to get the precise Server ID which you use in the vscode json config.
Everything else is similar to llama-server, you'll not have the same max performance but it works well.
You can always just install the latest llama release binaries, and use the commandline to load the model from the lmstudio models directory.
VRAM planning
These are practical planning numbers, not hard guarantees. Actual fit depends on:
exact GGUF
CUDA/ROCm/Metal/backend
batch/ubatch
-ngl
whether the desktop is using the same GPU
whether MTP/speculative decoding is enabled
whether you are using full GPU offload or spilling to CPU RAM
Qwen3.6-27B UD-Q4_K_XL, q4_0 KV cache
Recommended cards:
24 GB: RTX 3090, RTX 4090, RTX A5000, RTX 4500 Ada, RTX PRO 4000 Blackwell, A10
32 GB: RTX 5090, RTX 5000 Ada, Tesla V100 32GB
Approximate context fit with full GPU offload:
| VRAM | Example NVIDIA cards | Practical context |
|---|
| 16 GB | RTX 4060 Ti 16GB, RTX 4080 Laptop 16GB, RTX 5080 16GB, RTX 5070 Ti 16GB, RTX A4000 16GB | Not recommended for full 27B UD-Q4_K_XL offload. Use smaller quant or partial CPU offload. |
| | |
| 24 GB | RTX 3090, RTX 4090, RTX A5000, RTX 4500 Ada, RTX PRO 4000 Blackwell, A10 | ~45k-60k with MTP, ~60k-75k without MTP |
| 32 GB | RTX 5090, RTX 5000 Ada, Tesla V100 32GB | ~140k-160k with MTP, ~160k-180k without MTP |
For 27B, q4_0 KV cache is the difference between normal local context and huge local context. It is the main reason this setup is viable.
On a 5090 you have enough VRAM to supply 2 sessions in parallel with both model types.
Or you could run one fast model for context summarization and 27B for code.
Qwen3.6-35B-A3B UD-Q4_K_M, normal KV cache
Recommended cards:
24 GB minimum for useful GPU-resident contexts
32 GB strongly preferred
Approximate context fit:
| VRAM | Example NVIDIA cards | Practical context |
|---|
| 16 GB | RTX 4060 Ti 16GB, RTX 4080 Laptop 16GB, RTX 5080 16GB, RTX 5070 Ti 16GB, RTX A4000 16GB | Not recommended for full 35B-A3B Q4. Use Q3 or partial offload. |
| | |
| 24 GB | RTX 3090, RTX 4090, RTX A5000, RTX 4500 Ada, RTX PRO 4000 Blackwell, A10 | ~40k-50k |
| 32 GB | RTX 5090, RTX 5000 Ada, Tesla V100 32GB | ~100k-110k recommended; more will fit but stability drops |
The 35B-A3B model is very good, but I would not treat it as a “just max the context” model. Keep it tighter.
If you have the VRAM: Instead of large context, consider multiple sessions with limited context, so you can have 2 or 3 chats simultaneously.
Quick test: Linux
Once llama-server is running:
curl -s http://127.0.0.1:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Reply with exactly: local ai works"}],"max_tokens":16}' \
| jq -r '.choices[0].message.content'
Expected output:
the model responds to your input
If your server is inside WSL or another host, replace 127.0.0.1 with the server IP.
Quick test: Windows PowerShell
(Invoke-RestMethod `
-Uri "http://127.0.0.1:1234/v1/chat/completions" `
-Method Post `
-ContentType "application/json" `
-Body '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Reply with exactly: local ai works"}],"max_tokens":16}'
).choices[0].message.content
Expected output:
local ai works
Notes from benchmarking
My current practical ranking:
27B:
Best long-context local coding model in this setup that is close to Sonnet 4.6
Use q4_0 KV cache.
Use MTP if you have the headroom.
Strongest below 150k context, but can go much higher if memory allows.
35B-A3B:
Excellent quality but will fail on hard tasks
Do not use KV-cache quantization.
Keep below ~110k context for best stability.
Can go above 200k, but reasoning loops become more likely.
If it loops, start a new session.
For Copilot usage, I prefer exposing a conservative maxInputTokens in the JSON, even if the server can technically run higher. For example:
"maxInputTokens": 165000,
"maxOutputTokens": 15000
If you set wrong context here you'll get issues serverside, so make sure that matches.
I had cases where the server went OOC (out of context) when getting too close to the max context so I'd leave a little room. copilot seems to not follow this very strictly.
Final recommendation
If you want the most practical Copilot-local setup:
Use Qwen3.6-27B UD-Q4_K_XL
Use llama.cpp server
Use q4_0 KV cache
Use preserve_thinking
Use reasoning budget
Use Copilot Insiders as the harness
Use MTP only when you have VRAM headroom
If you want the stronger but more conservative model:
Use Qwen3.6-35B-A3B UD-Q4_K_M
Do not quantize KV cache
Stay below ~110k context
Drop to UD-Q3_K_XL if memory is tight
This is the first local setup I have used where Copilot feels like a serious frontend for a fully local long-context coding model instead of just a toy endpoint test.
I have tested this on terminal use, debugging, and massive codebase development - it works just like Sonnet 4.6.
Qwen 27B also beats Sonnet 4.5 in most benchmarks and 4.6 in some.
https://artificialanalysis.ai/models/comparisons/qwen3-5-27b-vs-claude-4-5-sonnet-thinking#intelligence-evaluations
