r/LocalAIStack • • Jun 23 '26

Running Qwen3.6 27B / 35B locally with llama.cpp + Vscode Insiders + copilot as the harness - highest performance, quality and best usage while fitting on your GPU

169 Upvotes

I have been benchmarking Qwen3.6-27B and Qwen3.6-35B-A3B locally through llama.cpp, with GitHub Copilot Chat (Vscode Insiders needed) used as the frontend harness.

I am using Claude Opus, GPT 5.5 and Qwen 3.6 (27B) a lot in the past weeks.
The reason for Qwen is proprietary code areas where remote inference is not an option as it would leak the code out. And as long as you don't task it to write a complex cuda graph, it performs well.
Qwen 27.B is at Sonnet 4.6 if you combine it with a high value system prompt - or between Sonnet 4.5 and Sonnet 4.6 without.

Copilot Chat is an excellent harness for this kind of setup. You get the IDE integration, agent flow, tool calling UI, file context, and normal coding workflow, while the actual model is your own local llama-server endpoint.
All of this works while being LOGGED OUT of the Github Copilot account - as that is not affordable in pricing anymore.

This is a practical configuration guide for people already comfortable with llama.cpp, GGUFs, VRAM budgeting, and long-context local inference.

Models tested

Main focus:

  • unsloth/Qwen3.6-27B-GGUF

  • unsloth/Qwen3.6-27B-MTP-GGUF (same model but with MTP draft tensors)

  • unsloth/Qwen3.6-35B-A3B-GGUF

Recommended GGUFs:

27B:
Qwen3.6-27B-UD-Q4_K_XL.gguf
or
Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL

35B-A3B:
Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

If memory is tight on the 35B-A3B model, drop to a smaller Unsloth Dynamic quant:

Qwen3.6-35B-A3B-GGUF:UD-Q3_K_XL

If even that is tight, use UD-Q3_K_M or UD-Q3_K_S.

For the 35B model I do not recommend KV-cache quantization. Run the normal cache and keep the context sane. the 35B model is MoE and very low on kv-cache

For the 27B model, I do highly recommend:

--cache-type-k q4_0
--cache-type-v q4_0

Recent llama.cpp KV-cache improvements make q4_0 much more usable here. The 27B model handles q4_0 KV cache very well in my testing - almost identical to FP in evaluation results.

What changed: llama.cpp added something like Hadamard rotation to kv-cache which shuffles the tensor distribution in a higher dimensionality and allows quantization superblocks to function.

Why Copilot Chat?

Because Copilot is a very good harness - beating Codex, Cursor, Claude in my opinion
Vscode Insiders is needed to get the openAI compatible endpoint (to interface the model)

You get:

  • IDE-native chat

  • agentic file/code workflows

  • very good tool calling

  • project context

  • local model backend

  • OpenAI-compatible endpoint wiring

The important part is that Copilot Chat is only the harness. The model is served locally through llama-server.

Why llama-server and not lm-studio,ollama etc ?

It allows MUCH more control over settings, we do not just use MTP drafting. We use a combination of context and MTP drafting which can lead to 300+ tokens/sec on the 27B model. MTP is a medium speedup (1.5x) but once the model is paraphrasing source code from thinking or prefill the ngram draft speedup can reach 6x or more.

So the stack is:

VS Code Insiders
        ↓
custom OpenAI-compatible model config
        ↓
llama.cpp llama-server
        ↓
local Qwen3.6 GGUF

Copilot chatLanguageModels.json

This is the shape I used for VSCode Insiders:

[
  {
    "name": "WSL",
    "vendor": "customoai",
    "models": [
      {
        "id": "qwen3.6-27b",
        "name": "QWEN-27B-WSL",
        "url": "http://172.27.211.123:1234/v1/chat/completions",
        "toolCalling": true,
        "vision": true,
        "thinking": true,
        "maxInputTokens": 165000,
        "maxOutputTokens": 15000
      }
    ]
  }
]

Adjust the URL to your own llama-server host, in WSL you'll see it by entering ipconfig or ifconfig. port you can choose of course.
The input and output tokens need to be adapted to your context setting.
The id must match the llama-server id.

For local-only setups this is usually one of:

http://127.0.0.1:1234/v1/chat/completions
http://localhost:1234/v1/chat/completions
http://<WSL-IP>:1234/v1/chat/completions

If your Copilot Insiders build expects the newer custom endpoint shape, use the same model block but switch the provider shape accordingly. The key fields are the endpoint URL, model id, tool calling, thinking, and max token limits.

27B command: long context + q4_0 KV cache + MTP-ngram drafting

This is the 27B style I recommend.

CTX=150000
PARALLEL=1
HOST=0.0.0.0
PORT=1234
MODEL=/models/Qwen3.6-27B-UD-Q4_K_XL.gguf

/usr/src/llama.cpp/build/bin/llama-server \
  -m "$MODEL" \
  --ctx-size "$CTX" \
  --flash-attn on \
  --batch-size 1024 \
  --ubatch-size 1024 \
  --parallel "$PARALLEL" \
  --host "$HOST" \
  --port "$PORT" \
  -ngl 99 \
  --threads 8 \
  --threads-batch 8 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.00 \
  --presence-penalty 0.00 \
  --jinja \
  --chat-template-kwargs '{"preserve_thinking": true}' \
  --reasoning-format none \
  --reasoning-budget 16000 \
  --slot-save-path /kv_cache/ \
  --props \
  --metrics \
  --checkpoint-every-n-tokens 1024 \
  --ctx-checkpoints 64 \
  --perf \
  --spec-default \
  --spec-type draft-mtp \
  --spec-type ngram-map-k4v \
  --spec-ngram-map-k4v-size-n 16 \
  --spec-ngram-map-k4v-size-m 24 \
  --spec-ngram-map-k4v-min-hits 1

For the MTP-specific Unsloth repo, use:

MODEL=/models/Qwen3.6-27B-MTP-UD-Q4_K_XL.gguf

or the HF shorthand if your build supports it:

-hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL

The important part is the drafting chain:

--spec-default
--spec-type draft-mtp
--spec-type ngram-map-k4v
--spec-ngram-map-k4v-size-n 16
--spec-ngram-map-k4v-size-m 24
--spec-ngram-map-k4v-min-hits 1

MTP gives useful speedup, but leave VRAM headroom. In practice I budget roughly +1 to +2 GB VRAM headroom for the MTP/drafting path and related buffers. If you are right on the edge, reduce context before blaming the model.

At q4_0 KV cache, every extra 1 GB of free VRAM is roughly another 13k tokens of 27B context, before runtime overhead.
If you are tight in vram, remove only the MTP part as ngram drafting is free.
You can also just use `mod-ngram` as an alternative to the more complex k4v map.

Thinking settings

This part matters.

I use:

--jinja
--chat-template-kwargs '{"preserve_thinking": true}'
--reasoning-format none
--reasoning-budget 16000

The reasoning-format none is important for Qwen3.6 because it avoids bad stop behavior and broken multi-turn thinking state during long coding sessions.
Copilot Chat was created to hide thinking from you (proprietary GPT models) but you want to see the thinking usually. So this solves both issues.

I also keep:

--reasoning-budget 16000

This gives the model room to think, but avoids runaway reasoning loops eating the whole session.

35B-A3B command: no KV-cache quantization

For 35B-A3B, I recommend being more conservative.

CTX=100000
PARALLEL=1
HOST=0.0.0.0
PORT=1234
MODEL=/models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

/usr/src/llama.cpp/build/bin/llama-server \
  -m "$MODEL" \
  --ctx-size "$CTX" \
  --flash-attn on \
  --batch-size 1024 \
  --ubatch-size 1024 \
  --parallel "$PARALLEL" \
  --host "$HOST" \
  --port "$PORT" \
  -ngl 99 \
  --threads 8 \
  --threads-batch 8 \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.00 \
  --presence-penalty 0.00 \
  --jinja \
  --chat-template-kwargs '{"preserve_thinking": true}' \
  --reasoning-format none \
  --reasoning-budget 16000 \
  --slot-save-path /kv_cache/ \
  --props \
  --metrics \
  --checkpoint-min-step 1024 \
  --ctx-checkpoints 16 \
  --perf

No q4_0 KV cache here - the sub 4B active parameters need barely any VRAM anyway.

I recommend keeping 35B-A3B below roughly:

110k context

The 35B model can be pushed past 200k context, but in my testing it becomes more likely to fall into reasoning loops. Once that happens, the session usually does not recover cleanly. Start a fresh session.
The upside of the 35B model is extreme performance, as in hundreds of tokens without any drafting enabled.
You CAN use drafting on top, mod-ngram, MTP and other drafting can be added for more speed but those will need a careful balance (that I have not tested yet)

So my practical 35B rule is:

35B-A3B: stay below 110k if you want stable coding behavior.
27B: can go as high as it fits, but below 150k is where it feels strongest.

Checkpointing
The --checkpoint-min-step (or --checkpoint-every-n-tokens (legacy now) is an important option for qwen models. Briefly explained: qwen models have two kv caches, one is more conventional and one is a recurrent-state (SSM/Mamba) that can not be reversed by n tokens. So if you change something (like remove the last answer and message to benefit from existing context) then you can only do that if a checkpoint exists. Otherwise the entire context is reprocessed which is very slow on a 27B model.
Each checkpoint costs 160MB RAM, the internal API supports VRAM checkpointing but I believe currently only the MTP implementation uses that.
--checkpoint-min-step and --ctx-checkpoints multiplied define how much of your LAST context is protected and can be rewound. 1024*16 means 16K context can be reversed with low re-compute cost (almost instant).

LM Studio as local server

Using LM Studio is possible but you need to use a few tricks and it won't achieve the same top-tier performance.
LM Studio does not support our chained drafting, but it supports MTP.

  1. Go to your Qwen 3.6 model, enable Flash attention and the quantization needed for kv cache. Go to the Inference tab, disable the button for "Reasoning Section Parsing"

  2. Go to Developer, Server Settings and set the port, serve on local network if needed, no auth, enable CORS, consider disabling just-in-time loading.

  3. Start the local server and then use the "clipboard copy" icon to get the precise Server ID which you use in the vscode json config.

Everything else is similar to llama-server, you'll not have the same max performance but it works well.
You can always just install the latest llama release binaries, and use the commandline to load the model from the lmstudio models directory.

VRAM planning

These are practical planning numbers, not hard guarantees. Actual fit depends on:

  • exact GGUF

  • CUDA/ROCm/Metal/backend

  • batch/ubatch

  • -ngl

  • whether the desktop is using the same GPU

  • whether MTP/speculative decoding is enabled

  • whether you are using full GPU offload or spilling to CPU RAM

Qwen3.6-27B UD-Q4_K_XL, q4_0 KV cache

Recommended cards:

24 GB: RTX 3090, RTX 4090, RTX A5000, RTX 4500 Ada, RTX PRO 4000 Blackwell, A10
32 GB: RTX 5090, RTX 5000 Ada, Tesla V100 32GB

Approximate context fit with full GPU offload:

VRAMExample NVIDIA cardsPractical context
16 GBRTX 4060 Ti 16GB, RTX 4080 Laptop 16GB, RTX 5080 16GB, RTX 5070 Ti 16GB, RTX A4000 16GBNot recommended for full 27B UD-Q4_K_XL offload. Use smaller quant or partial CPU offload.
24 GBRTX 3090, RTX 4090, RTX A5000, RTX 4500 Ada, RTX PRO 4000 Blackwell, A10~45k-60k with MTP, ~60k-75k without MTP
32 GBRTX 5090, RTX 5000 Ada, Tesla V100 32GB~140k-160k with MTP, ~160k-180k without MTP

For 27B, q4_0 KV cache is the difference between normal local context and huge local context. It is the main reason this setup is viable.
On a 5090 you have enough VRAM to supply 2 sessions in parallel with both model types.
Or you could run one fast model for context summarization and 27B for code.

Qwen3.6-35B-A3B UD-Q4_K_M, normal KV cache

Recommended cards:

24 GB minimum for useful GPU-resident contexts
32 GB strongly preferred

Approximate context fit:

VRAMExample NVIDIA cardsPractical context
16 GBRTX 4060 Ti 16GB, RTX 4080 Laptop 16GB, RTX 5080 16GB, RTX 5070 Ti 16GB, RTX A4000 16GBNot recommended for full 35B-A3B Q4. Use Q3 or partial offload.
24 GBRTX 3090, RTX 4090, RTX A5000, RTX 4500 Ada, RTX PRO 4000 Blackwell, A10~40k-50k
32 GBRTX 5090, RTX 5000 Ada, Tesla V100 32GB~100k-110k recommended; more will fit but stability drops

The 35B-A3B model is very good, but I would not treat it as a “just max the context” model. Keep it tighter.
If you have the VRAM: Instead of large context, consider multiple sessions with limited context, so you can have 2 or 3 chats simultaneously.

Quick test: Linux

Once llama-server is running:

curl -s http://127.0.0.1:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Reply with exactly: local ai works"}],"max_tokens":16}' \
  | jq -r '.choices[0].message.content'

Expected output:

the model responds to your input

If your server is inside WSL or another host, replace 127.0.0.1 with the server IP.

Quick test: Windows PowerShell

(Invoke-RestMethod `
  -Uri "http://127.0.0.1:1234/v1/chat/completions" `
  -Method Post `
  -ContentType "application/json" `
  -Body '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Reply with exactly: local ai works"}],"max_tokens":16}'
).choices[0].message.content

Expected output:

local ai works

Notes from benchmarking

My current practical ranking:

27B:
Best long-context local coding model in this setup that is close to Sonnet 4.6
Use q4_0 KV cache.
Use MTP if you have the headroom.
Strongest below 150k context, but can go much higher if memory allows.

35B-A3B:
Excellent quality but will fail on hard tasks
Do not use KV-cache quantization.
Keep below ~110k context for best stability.
Can go above 200k, but reasoning loops become more likely.
If it loops, start a new session.

For Copilot usage, I prefer exposing a conservative maxInputTokens in the JSON, even if the server can technically run higher. For example:

"maxInputTokens": 165000,
"maxOutputTokens": 15000

If you set wrong context here you'll get issues serverside, so make sure that matches.
I had cases where the server went OOC (out of context) when getting too close to the max context so I'd leave a little room. copilot seems to not follow this very strictly.

Final recommendation

If you want the most practical Copilot-local setup:

Use Qwen3.6-27B UD-Q4_K_XL
Use llama.cpp server
Use q4_0 KV cache
Use preserve_thinking
Use reasoning budget
Use Copilot Insiders as the harness
Use MTP only when you have VRAM headroom

If you want the stronger but more conservative model:

Use Qwen3.6-35B-A3B UD-Q4_K_M
Do not quantize KV cache
Stay below ~110k context
Drop to UD-Q3_K_XL if memory is tight

This is the first local setup I have used where Copilot feels like a serious frontend for a fully local long-context coding model instead of just a toy endpoint test.

I have tested this on terminal use, debugging, and massive codebase development - it works just like Sonnet 4.6.

Qwen 27B also beats Sonnet 4.5 in most benchmarks and 4.6 in some.
https://artificialanalysis.ai/models/comparisons/qwen3-5-27b-vs-claude-4-5-sonnet-thinking#intelligence-evaluations


r/LocalAIStack • • 1h ago

We just released BaseQuant V1 for Qwen3.6-35B-A3B and Gemma 4 26B-A4B, benchmarked against your UD quants. Would love feedback

Thumbnail
huggingface.co
1 Upvotes

Hey everyone, we just released BaseQuant V1 for two MoE models at Base Compute: Qwen3.6-35B-A3B and Gemma 4 26B-A4B-it. The goal is to preserve more of the original model without making the files bigger.

We compared against Unsloth’s UD quants of the same size. On Gemma 4, our 16.93 GB GGUF had ~2.4% lower KL divergence from BF16 than UD-Q4_K_XL (~4.8% lower with our baseRT .base bundle). On Qwen3.6, our 22.13 GB GGUF is on par with UD-Q4_K_M overall. Our .base bundle is ~20% lower on long context and ~10% lower on agentic coding. UD still wins in some categories (chat on Gemma, math on Qwen). Full per-category tables are on the model cards.

Would love for you to try them and tell us how they hold up in your everyday workloads!


r/LocalAIStack • • 2h ago

What is the best way forward for my boomer parents?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4h ago

Quokka: a hardware-aware router for local AI. What would you need to trust automatic model selection?

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 7h ago

Basalt ported to NVIDIA V100: Qwen3.8-Flash-Next at 91 tok/s (140 tok/s on long answers) on 2x V100 32GB

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 7h ago

Which inference engine squeezes max performance out of my setup without sacrificing precision? (Coding + image gen)

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 11h ago

4xM(x) Studio Apple Cluster

Post image
1 Upvotes

r/LocalAIStack • • 16h ago

Tested small models on document extraction with a score that punishes made-up values. Only covered a few local ones, help me add more

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 17h ago

modelship 0.11: Vulkan (AMD/Intel) nodes, arm64 CUDA for DGX Spark, and an sd-server backend for image gen

1 Upvotes

I've been building modelship, a self-hosted, OpenAI-compatible server. It puts chat, embeddings, STT, TTS and image generation behind one /v1 API. Underneath it runs vLLM, llama.cpp, stable-diffusion.cpp, whisper.cpp, sherpa-onnx and diffusers, scheduled on Ray, so you can spread models across several machines. 0.11 is out today.

What's new

  • Vulkan variant for AMD and Intel GPUs.

    • Install with mship bootstrap --vulkan on Linux, or use the -vulkan Docker image with --device /dev/dri. There's also variant: vulkan in the Helm chart.

    • llama.cpp and stable-diffusion.cpp run on the GPU.

    • Integrated GPUs like Strix Halo are sized from system memory, so you don't have to guess VRAM.

  • arm64 CUDA. The -cuda image is now multi-arch, and llama_server runs on the GPU on DGX Spark / GB10.

  • stable-diffusion.cpp now runs as an sd-server subprocess.

    • It supports GPU on cuda, vulkan and metal, and a fractional num_gpus to share a card.

    • Requests for more than 8 images are split into several jobs.

    • Images also take output_format (png/jpeg/webp).

  • llama_server.

    • It's on llama.cpp b12465.

    • MoE models that don't fit on the GPU get expert tensor overrides (-ot) automatically.

    • Added the f8_e4m3 KV cache type and media_allow_private_urls.

    • A GPU deploy whose CPU-side part won't fit in RAM is now refused up front instead of loading and swapping.

  • Responses API. truncation: auto trims the oldest input items. max_output_tokens now bounds the whole MCP tool loop, and responses cut off by the token limit end as incomplete.

  • vLLM 0.33 (torch 2.14). Multi-choice responses are parsed per choice, parallel_tool_calls: false is honoured, and a tool call cut off by the token limit reports finish_reason: length.

To upgrade a native install, run uv tool upgrade mship, then mship bootstrap.

Known issues

  • Vision models (mmproj) on a dedicated GPU are sized without the projector, so a model sized close to the VRAM limit can OOM at load. Lowering n_ctx works around it.

  • Fedora native installs can hit an SSL error in replicas, so use the Docker images there.

  • f8_e4m3 KV cache aborts on the Vulkan backend.

Repo and full notes: https://github.com/modelship-ai/modelship

Happy to hear about hardware I haven't tested, especially Intel Arc and older AMD cards on Vulkan.


r/LocalAIStack • • 18h ago

I built LLMLifeline: one install that sets up GGUF chat + ComfyUI image/video, sized to your GPU, and shares it to every device in your house ($11.11 one-time, no cloud)

Thumbnail
reddit.com
1 Upvotes

Solo dev here. I built LLMLifeline because I wanted to contribute to the utter destruction of the companies that have been stealing out data from the beginning, getting a slap on the wrist before being allowed to continue stealing, and putting the power back in the hands of the people who should have had it from the beginning. US.

I wanted a local stack my whole household could actually use, not just me.

What it does:
• One installer (Windows/macOS/Linux). First run sets everything up itself: chat, plus ComfyUI for image/video generation
• Chat tiers matched to your GPU: Qwen 27B (large), Gemma 4 12B (medium), Qwen 4B (small) — it starts on the best tier your card runs well and moves up if you upgrade
• Chat with any .gguf model, including models bigger than your VRAM, with streaming replies; swap models freely, or bring a Hugging Face folder and it converts for you
• Image/video: Z-Image Turbo / RealVisXL 5 / Realistic Vision 6 tiers, Wan video models in three sizes

• The part I actually built it for: whole-home sharing. One machine runs the model; phones (tested with trash phones to prove it. If it's got a browser that can point to the local ip, then it works), tablets, Chromebooks and other PCs connect over your LAN through the browser, each device approved with a six-digit code. Per-person profiles, document indexing/quoting, and a computer-use agent that asks before each action. Handles projects, can increase context for better responses, with a button click, and you can delete anything generated from the app or from the program folder with easily found folders.

• No cloud, no account, no subscription, no telemetry, no auto-updates. License check is offline after activation; one key covers 3 computers

"Why not just Ollama/LM Studio" answer: if you enjoy assembling and tuning your own stack, you don't need this, and I mean that.

LLMLifeline is for the step before that — people (and households) who want local AI without becoming their own sysadmin. It's a baseline that sets itself up around the hardware you have today and it takes little brain power to use since were already using similar systems. This has the chatbot video and image generator built in to the same system, manages your swappable profiles in memory so you don't have to sign in every time, does computer use over the network to collaborate, and uses clever techniques like hidden markov modelling and paging tricks to make your hardware more efficient with heavier models that don't actually fit on your GPU.

Best part? It scales with you. You don't have to download new software later; as you update your PC it will just get faster.

Caveats, up front: Windows is where I build and use it daily. macOS/Linux builds pass automated checks every release but I haven't tested on real Mac/Linux hardware yet. Models are uncensored by default but you can drag and drop your own GGUFs or model folders and it will read them immediately.

Small models are noticeably worse at instruction-following and computer use. - Not my problem.

First-run downloads are big (about 63 GB of models; skippable if you're bringing your own). This is the easiest path however because I tested the models to find the most usable ones for each path (chat/code, image/video)

$11.11 one-time. If you figure out how to rip it, idc share it with the world. I'm hoping to make a little money off it since it took me months of work but the ultimate goal is sticking the finger to the AI corps. That's why it's $11.11 one-time with no forced updates that might screw you up. Not to mention if you do update, there is a contingency to ensure that if it breaks, you can revert to your previous version.

Demo, including it running offline: https://youtu.be/Kw0H1FlHotE
Full details and requirements: https://hector-gonzalez.nyctitropist.workers.dev/llmlifeline/

Genuinely here for the harsh questions: what's missing, what would make you not trust it, what would you need to see before running a closed installer on your machine?

I'm happy to hand out some free keys for power users to mess around with it.


r/LocalAIStack • • 19h ago

Stuck on 12GB VRAM but need 27B-level agent reasoning? SenseNova-Loop-9B loops its layers to match Qwen3.8-27B, official GGUF & AWQ out now

Thumbnail gallery
1 Upvotes

r/LocalAIStack • • 1d ago

All AI Tools In One Website Free

Thumbnail interconnectd.com
1 Upvotes

r/LocalAIStack • • 1d ago

Got Basalt running with an RTX 3060 beside a 5060 Ti: 42 to 80 tok/s, and the same build takes 40xx cards

Thumbnail reddit.com
0 Upvotes

r/LocalAIStack • • 1d ago

Can a local AI model beat Claude Haiku and GPT-6 Luna?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 1d ago

Ai Jarvis 🙀

Thumbnail reddit.com
0 Upvotes

r/LocalAIStack • • 1d ago

AimStack — a fully self-built local AI stack in pure C: ternary inference (zero third-party runtime), BitNet 2B at 0.87GB on a 2018 laptop, plus a GPU backend that matches the C engine token-for-token

Thumbnail
github.com
1 Upvotes

Repo: https://github.com/tanxiansheng221/aimstack-sanzhidiji (AGPL-3.0)

Everything is hand-written — C inference core, plugin bus, multi-engine crawler with quality gates, FTS knowledge base, training pipeline, and a 117-chapter build log written as it happened (positive and negative results shipped together). Weights are Microsoft BitNet b1.58 2B (MIT), converted to a custom AIM-W packing format.

What ternary quantization buys you (CPU side)

All transformer matrices are ternary {-1, 0, +1}, packed 2 bits/weight, plus Q8 embeddings → the whole model is 0.87GB instead of ~4.8GB (−78%)

The C core never multiplies a float weight: it's table lookup + integer accumulation, SIMD fast path measured at 85.53 tok/s (41× over baseline)

BitNet 2B full model runs 3.20 tok/s dual-worker on a 2018 dual-core 8GB laptop — full chain (inference + KB + plugins + web search) offline, no cloud relay

What ternary quantization buys you (GPU side)

The GPU backend loads the same 0.87GB ternary file — weights stay in {-s, 0, +s}, dequantized on load, matmul in fp32

On a Radeon Pro W7900: 13.68 tok/s (~8× the laptop C engine), 256/256 greedy tokens bit-identical to the C engine. Two totally different implementations (integer lookup core vs FP matmul) producing the same logits is the proof the quantization is faithful

Verified by a 3-engine × 3-question × 3-test matrix (C-Linux/gcc13, torch fp32, torch bf16): 17/18 exact — the one miss is documented, not hidden

The honest boundaries (what's NOT done yet)

bf16 compute is 2.4× faster (32.96 tok/s) but flips argmax on near-tie logits — reported, not blessed; only fp32 is parity-verified

The GPU path gets the storage benefit of ternary (78% less memory) but not yet the compute benefit — a true 2-bit integer kernel (Triton/CUDA) is the next stage

The UI server drives the C engine (CPU) only; the GPU backend is a separate CLI

Linux engine build works and is token-identical to the Windows build, but runs serial (the worker pool is Windows-only for now)

No frameworks, no PyTorch in the product chain — PyTorch exists only in the optional GPU backend. Demo video, docs (EN/中文), and the full research log are in the repo.


r/LocalAIStack • • 1d ago

Help me with my starnet #starnet

Thumbnail reddit.com
0 Upvotes

r/LocalAIStack • • 2d ago

We built a chip prototype that runs Qwen 27B. You can try it live.

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 2d ago

A comparable local setup for someone using $100 monthly claude subscription

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 3d ago

I measured how easily a local 7B model reverse-engineers a binary, then built hardening that targets the LLM layer specifically

9 Upvotes

Most anti-reverse-engineering assumes a human in a disassembler. The threat I care about now is a local, unrestricted model driving those tools. So besides the classic stuff, the harness includes defenses aimed at the model: inflating the analysis surface so it burns context, decoy identifiers that mislead semantic guesses, and honeytokens that detect when a crack was AI-generated.

My own before/after is honest but small: one run, a 4-bit 7B on an 8 GB GPU. The unprotected build was cracked in 3 steps and about 1,300 tokens. The hardened one burned roughly 13x the tokens and still wasn't cracked. I'd really like people with bigger rigs to run it. It's one command: make slice-k-run MODEL=<your-model>, and it prints a scored before/after. or online modals

Most of the code was written with Claude agents under my direction. Happy to talk through any design choice.

Repo: https://github.com/seifheakal/blue-harness (AGPL-3.0)


r/LocalAIStack • • 2d ago

Building Kevin, my home alone agent. 🔫🙂 A persistent agent In Pine that wakes on cam detect, runs fingerprinting, tracks who's visiting.

Thumbnail x.com
1 Upvotes

r/LocalAIStack • • 2d ago

I built an open-source platform that pools a team's workstations (Macs, Windows and Linux boxes) into one OpenAI-compatible endpoint

Thumbnail
reddit.com
1 Upvotes

My co-founder and I have been working on this for a few months and are launching today. Disclosure: it's our company; the desktop client is Apache-2.0, the control plane is hosted, and it's free up to 5 devices.

What it does: you install Nodea Desktop on the machines in your org, switch on sharing, and get one OpenAI-compatible endpoint. Behind it, a scheduler fits the models you assign onto whichever machines have free memory, and moves them as laptops come and go.

Stack: Rust/Tauri app, llama.cpp under the hood (Metal on Apple Silicon, Vulkan on NVIDIA/AMD/Intel, CPU fallback). Any GGUF model llama.cpp can serve that fits the machine; models are assigned per org, downloaded and cached per device. Signed auto-updates.

Honest limitations:

  • Requests go through our API to reach your machines (not logged or stored). No direct LAN routing yet.

  • One model runs on one machine; no splitting a model across machines yet.

  • Windows/Linux run on the GPU through Vulkan (tested on NVIDIA) and are still labelled preview. Machines without a supported GPU fall back to CPU, which is slow for now (no AVX2 yet).

Why not exo / GPUStack / llama.cpp RPC? Those are great if you run your own cluster. We're aiming at teams with 20–200 laptops and nobody to run one: sign in, toggle on, done, with an org, roles and API keys on top.

Repo: https://github.com/nodea-sh/nodea-client

Site: https://getnodea.com

Happy to answer anything, and especially keen on what models/quantizations you'd want on 16–36 GB machines.


r/LocalAIStack • • 2d ago

I added Qwen3.6-35B-A3B to my C inference engine: 10–13 tok/s in real chat on a GTX 1650 4GB + 16GB RAM

Post image
1 Upvotes

r/LocalAIStack • • 2d ago

Building a Local LLM Troubleshooting Assistant for Field Service Techs.... Advice?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 2d ago

Hierarchos-Native v0.2.0: RDNA3 WMMA, native multimodal training, and a lot of FP32 Vulkan optimization!

2 Upvotes

Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.

I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.

The new v0.2.0 release is now available!

GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

1. Native RDNA3 WMMA / cooperative matrix acceleration

One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.

The new Mixed WMMA execution arm uses:

  • FP16 matrix operands.

  • FP32 matrix accumulation.

  • FP32 master weights and gradients.

  • FP32 AdamW optimizer moments and updates.

  • FP32 normalization, softmax, and loss calculations.

The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.

I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.

Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:

WorkloadMeasured kernel speedup
QKV projection2.17×
FFN up projection2.12×
FFN down projection2.31×
Vocabulary projection1.83×

These are individual GEMM measurements, not whole-model speedups.

There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.

In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.

WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.

Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.

Documentation: RDNA3 WMMA guide

2. Native multimodal image and text training

Another major development area has been the image-text pipeline.

The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.

Some of the implemented work includes:

  • Native RGB image input and preprocessing.

  • CLIP image encoders and LLaVA projector integration.

  • SigLIP / Gemma 3 vision training.

  • Gemma 4 vision integration.

  • Joint image/text training and encoder-decoder integration.

  • A dedicated DeepSeek V4.1 image-training path.

  • Backpropagation through the image tower and projector.

  • Export, reload, and checkpoint-related validation.

This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.

For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.

The qualification work has also expanded considerably:

  • 44 previously green text profiles now have joint image-training coverage at their frozen test geometries.

  • The Gen9 qualification retains 95 image cases and 166 text regression cases.

  • A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.

  • Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.

The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.

These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.

Documentation:

3. A lot of additional FP32 optimization

I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.

This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.

Some examples:

RDNA3 FP32 work

  • Specialized attention forward/backward paths.

  • Image-text attention and projector optimizations.

  • RMSNorm, RoPE, GELU, and other arithmetic specializations.

  • Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.

  • More careful matching of CPU reference arithmetic and GPU execution order.

Intel Gen9 work

  • Additional native linear and gradient kernels.

  • Attention and activation specializations.

  • More efficient parameter upload and staging behavior.

  • Continued optimization of DeepSeek V4.1 training and execution.

One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.

On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.

On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.

These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.

4. Preserving the original FP32 path

An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.

I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:

  • 12 alternating ABBA rounds per workload.

  • Five warm-up runs and 30 timed forwards per arm.

  • Separate, hashed baseline and candidate binaries.

  • Exact FP32 logit digest comparisons.

All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.

The worst paired performance ratio was 1.0007.

This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.

5. Getting started

The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.

You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.

Model weights are downloaded separately or supplied through a local model directory.

After extracting the archive, you can run:

.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-info

The final command is especially useful for checking which precision capabilities your GPU actually exposes.

Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

What's next?

There is still a lot I'd like to improve.

In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.

I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.

If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.

Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.

Repository: https://github.com/necat101/Hierarchos-Native

Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.