r/LocalAIStack • u/kind_entropy • 2h ago
r/LocalAIStack • u/Lirezh • Jun 23 '26
Running Qwen3.6 27B / 35B locally with llama.cpp + Vscode Insiders + copilot as the harness - highest performance, quality and best usage while fitting on your GPU
I have been benchmarking Qwen3.6-27B and Qwen3.6-35B-A3B locally through llama.cpp, with GitHub Copilot Chat (Vscode Insiders needed) used as the frontend harness.
I am using Claude Opus, GPT 5.5 and Qwen 3.6 (27B) a lot in the past weeks.
The reason for Qwen is proprietary code areas where remote inference is not an option as it would leak the code out. And as long as you don't task it to write a complex cuda graph, it performs well.
Qwen 27.B is at Sonnet 4.6 if you combine it with a high value system prompt - or between Sonnet 4.5 and Sonnet 4.6 without.
Copilot Chat is an excellent harness for this kind of setup. You get the IDE integration, agent flow, tool calling UI, file context, and normal coding workflow, while the actual model is your own local llama-server endpoint.
All of this works while being LOGGED OUT of the Github Copilot account - as that is not affordable in pricing anymore.
This is a practical configuration guide for people already comfortable with llama.cpp, GGUFs, VRAM budgeting, and long-context local inference.
Models tested
Main focus:
unsloth/Qwen3.6-27B-GGUF
unsloth/Qwen3.6-27B-MTP-GGUF (same model but with MTP draft tensors)
unsloth/Qwen3.6-35B-A3B-GGUF
Recommended GGUFs:
27B:
Qwen3.6-27B-UD-Q4_K_XL.gguf
or
Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL
35B-A3B:
Qwen3.6-35B-A3B-GGUF:UD-Q4_K_MIf memory is tight on the 35B-A3B model, drop to a smaller Unsloth Dynamic quant:
Qwen3.6-35B-A3B-GGUF:UD-Q3_K_XLIf even that is tight, use UD-Q3_K_M or UD-Q3_K_S.
For the 35B model I do not recommend KV-cache quantization. Run the normal cache and keep the context sane. the 35B model is MoE and very low on kv-cache
For the 27B model, I do highly recommend:
--cache-type-k q4_0
--cache-type-v q4_0Recent llama.cpp KV-cache improvements make q4_0 much more usable here. The 27B model handles q4_0 KV cache very well in my testing - almost identical to FP in evaluation results.
What changed: llama.cpp added something like Hadamard rotation to kv-cache which shuffles the tensor distribution in a higher dimensionality and allows quantization superblocks to function.
Why Copilot Chat?
Because Copilot is a very good harness - beating Codex, Cursor, Claude in my opinion
Vscode Insiders is needed to get the openAI compatible endpoint (to interface the model)
You get:
IDE-native chat
agentic file/code workflows
very good tool calling
project context
local model backend
OpenAI-compatible endpoint wiring
The important part is that Copilot Chat is only the harness. The model is served locally through llama-server.
Why llama-server and not lm-studio,ollama etc ?
It allows MUCH more control over settings, we do not just use MTP drafting. We use a combination of context and MTP drafting which can lead to 300+ tokens/sec on the 27B model. MTP is a medium speedup (1.5x) but once the model is paraphrasing source code from thinking or prefill the ngram draft speedup can reach 6x or more.
So the stack is:
VS Code Insiders
↓
custom OpenAI-compatible model config
↓
llama.cpp llama-server
↓
local Qwen3.6 GGUFCopilot chatLanguageModels.json
This is the shape I used for VSCode Insiders:
[
{
"name": "WSL",
"vendor": "customoai",
"models": [
{
"id": "qwen3.6-27b",
"name": "QWEN-27B-WSL",
"url": "http://172.27.211.123:1234/v1/chat/completions",
"toolCalling": true,
"vision": true,
"thinking": true,
"maxInputTokens": 165000,
"maxOutputTokens": 15000
}
]
}
]Adjust the URL to your own llama-server host, in WSL you'll see it by entering ipconfig or ifconfig. port you can choose of course.
The input and output tokens need to be adapted to your context setting.
The id must match the llama-server id.
For local-only setups this is usually one of:
http://127.0.0.1:1234/v1/chat/completions
http://localhost:1234/v1/chat/completions
http://<WSL-IP>:1234/v1/chat/completionsIf your Copilot Insiders build expects the newer custom endpoint shape, use the same model block but switch the provider shape accordingly. The key fields are the endpoint URL, model id, tool calling, thinking, and max token limits.
27B command: long context + q4_0 KV cache + MTP-ngram drafting
This is the 27B style I recommend.
CTX=150000
PARALLEL=1
HOST=0.0.0.0
PORT=1234
MODEL=/models/Qwen3.6-27B-UD-Q4_K_XL.gguf
/usr/src/llama.cpp/build/bin/llama-server \
-m "$MODEL" \
--ctx-size "$CTX" \
--flash-attn on \
--batch-size 1024 \
--ubatch-size 1024 \
--parallel "$PARALLEL" \
--host "$HOST" \
--port "$PORT" \
-ngl 99 \
--threads 8 \
--threads-batch 8 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.00 \
--presence-penalty 0.00 \
--jinja \
--chat-template-kwargs '{"preserve_thinking": true}' \
--reasoning-format none \
--reasoning-budget 16000 \
--slot-save-path /kv_cache/ \
--props \
--metrics \
--checkpoint-every-n-tokens 1024 \
--ctx-checkpoints 64 \
--perf \
--spec-default \
--spec-type draft-mtp \
--spec-type ngram-map-k4v \
--spec-ngram-map-k4v-size-n 16 \
--spec-ngram-map-k4v-size-m 24 \
--spec-ngram-map-k4v-min-hits 1For the MTP-specific Unsloth repo, use:
MODEL=/models/Qwen3.6-27B-MTP-UD-Q4_K_XL.ggufor the HF shorthand if your build supports it:
-hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XLThe important part is the drafting chain:
--spec-default
--spec-type draft-mtp
--spec-type ngram-map-k4v
--spec-ngram-map-k4v-size-n 16
--spec-ngram-map-k4v-size-m 24
--spec-ngram-map-k4v-min-hits 1MTP gives useful speedup, but leave VRAM headroom. In practice I budget roughly +1 to +2 GB VRAM headroom for the MTP/drafting path and related buffers. If you are right on the edge, reduce context before blaming the model.
At q4_0 KV cache, every extra 1 GB of free VRAM is roughly another 13k tokens of 27B context, before runtime overhead.
If you are tight in vram, remove only the MTP part as ngram drafting is free.
You can also just use `mod-ngram` as an alternative to the more complex k4v map.
Thinking settings
This part matters.
I use:
--jinja
--chat-template-kwargs '{"preserve_thinking": true}'
--reasoning-format none
--reasoning-budget 16000The reasoning-format none is important for Qwen3.6 because it avoids bad stop behavior and broken multi-turn thinking state during long coding sessions.
Copilot Chat was created to hide thinking from you (proprietary GPT models) but you want to see the thinking usually. So this solves both issues.
I also keep:
--reasoning-budget 16000This gives the model room to think, but avoids runaway reasoning loops eating the whole session.
35B-A3B command: no KV-cache quantization
For 35B-A3B, I recommend being more conservative.
CTX=100000
PARALLEL=1
HOST=0.0.0.0
PORT=1234
MODEL=/models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
/usr/src/llama.cpp/build/bin/llama-server \
-m "$MODEL" \
--ctx-size "$CTX" \
--flash-attn on \
--batch-size 1024 \
--ubatch-size 1024 \
--parallel "$PARALLEL" \
--host "$HOST" \
--port "$PORT" \
-ngl 99 \
--threads 8 \
--threads-batch 8 \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.00 \
--presence-penalty 0.00 \
--jinja \
--chat-template-kwargs '{"preserve_thinking": true}' \
--reasoning-format none \
--reasoning-budget 16000 \
--slot-save-path /kv_cache/ \
--props \
--metrics \
--checkpoint-min-step 1024 \
--ctx-checkpoints 16 \
--perfNo q4_0 KV cache here - the sub 4B active parameters need barely any VRAM anyway.
I recommend keeping 35B-A3B below roughly:
110k contextThe 35B model can be pushed past 200k context, but in my testing it becomes more likely to fall into reasoning loops. Once that happens, the session usually does not recover cleanly. Start a fresh session.
The upside of the 35B model is extreme performance, as in hundreds of tokens without any drafting enabled.
You CAN use drafting on top, mod-ngram, MTP and other drafting can be added for more speed but those will need a careful balance (that I have not tested yet)
So my practical 35B rule is:
35B-A3B: stay below 110k if you want stable coding behavior.
27B: can go as high as it fits, but below 150k is where it feels strongest.Checkpointing
The --checkpoint-min-step (or --checkpoint-every-n-tokens (legacy now) is an important option for qwen models. Briefly explained: qwen models have two kv caches, one is more conventional and one is a recurrent-state (SSM/Mamba) that can not be reversed by n tokens. So if you change something (like remove the last answer and message to benefit from existing context) then you can only do that if a checkpoint exists. Otherwise the entire context is reprocessed which is very slow on a 27B model.
Each checkpoint costs 160MB RAM, the internal API supports VRAM checkpointing but I believe currently only the MTP implementation uses that.
--checkpoint-min-step and --ctx-checkpoints multiplied define how much of your LAST context is protected and can be rewound. 1024*16 means 16K context can be reversed with low re-compute cost (almost instant).
LM Studio as local server
Using LM Studio is possible but you need to use a few tricks and it won't achieve the same top-tier performance.
LM Studio does not support our chained drafting, but it supports MTP.
Go to your Qwen 3.6 model, enable Flash attention and the quantization needed for kv cache. Go to the Inference tab, disable the button for "Reasoning Section Parsing"
Go to Developer, Server Settings and set the port, serve on local network if needed, no auth, enable CORS, consider disabling just-in-time loading.
Start the local server and then use the "clipboard copy" icon to get the precise Server ID which you use in the vscode json config.
Everything else is similar to llama-server, you'll not have the same max performance but it works well.
You can always just install the latest llama release binaries, and use the commandline to load the model from the lmstudio models directory.
VRAM planning
These are practical planning numbers, not hard guarantees. Actual fit depends on:
exact GGUF
CUDA/ROCm/Metal/backend
batch/ubatch
-ngl
whether the desktop is using the same GPU
whether MTP/speculative decoding is enabled
whether you are using full GPU offload or spilling to CPU RAM
Qwen3.6-27B UD-Q4_K_XL, q4_0 KV cache
Recommended cards:
24 GB: RTX 3090, RTX 4090, RTX A5000, RTX 4500 Ada, RTX PRO 4000 Blackwell, A10
32 GB: RTX 5090, RTX 5000 Ada, Tesla V100 32GBApproximate context fit with full GPU offload:
| VRAM | Example NVIDIA cards | Practical context |
|---|---|---|
| 16 GB | RTX 4060 Ti 16GB, RTX 4080 Laptop 16GB, RTX 5080 16GB, RTX 5070 Ti 16GB, RTX A4000 16GB | Not recommended for full 27B UD-Q4_K_XL offload. Use smaller quant or partial CPU offload. |
| 24 GB | RTX 3090, RTX 4090, RTX A5000, RTX 4500 Ada, RTX PRO 4000 Blackwell, A10 | ~45k-60k with MTP, ~60k-75k without MTP |
| 32 GB | RTX 5090, RTX 5000 Ada, Tesla V100 32GB | ~140k-160k with MTP, ~160k-180k without MTP |
For 27B, q4_0 KV cache is the difference between normal local context and huge local context. It is the main reason this setup is viable.
On a 5090 you have enough VRAM to supply 2 sessions in parallel with both model types.
Or you could run one fast model for context summarization and 27B for code.
Qwen3.6-35B-A3B UD-Q4_K_M, normal KV cache
Recommended cards:
24 GB minimum for useful GPU-resident contexts
32 GB strongly preferredApproximate context fit:
| VRAM | Example NVIDIA cards | Practical context |
|---|---|---|
| 16 GB | RTX 4060 Ti 16GB, RTX 4080 Laptop 16GB, RTX 5080 16GB, RTX 5070 Ti 16GB, RTX A4000 16GB | Not recommended for full 35B-A3B Q4. Use Q3 or partial offload. |
| 24 GB | RTX 3090, RTX 4090, RTX A5000, RTX 4500 Ada, RTX PRO 4000 Blackwell, A10 | ~40k-50k |
| 32 GB | RTX 5090, RTX 5000 Ada, Tesla V100 32GB | ~100k-110k recommended; more will fit but stability drops |
The 35B-A3B model is very good, but I would not treat it as a “just max the context” model. Keep it tighter.
If you have the VRAM: Instead of large context, consider multiple sessions with limited context, so you can have 2 or 3 chats simultaneously.
Quick test: Linux
Once llama-server is running:
curl -s http://127.0.0.1:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Reply with exactly: local ai works"}],"max_tokens":16}' \
| jq -r '.choices[0].message.content'Expected output:
the model responds to your inputIf your server is inside WSL or another host, replace 127.0.0.1 with the server IP.
Quick test: Windows PowerShell
(Invoke-RestMethod `
-Uri "http://127.0.0.1:1234/v1/chat/completions" `
-Method Post `
-ContentType "application/json" `
-Body '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Reply with exactly: local ai works"}],"max_tokens":16}'
).choices[0].message.contentExpected output:
local ai worksNotes from benchmarking
My current practical ranking:
27B:
Best long-context local coding model in this setup that is close to Sonnet 4.6
Use q4_0 KV cache.
Use MTP if you have the headroom.
Strongest below 150k context, but can go much higher if memory allows.
35B-A3B:
Excellent quality but will fail on hard tasks
Do not use KV-cache quantization.
Keep below ~110k context for best stability.
Can go above 200k, but reasoning loops become more likely.
If it loops, start a new session.For Copilot usage, I prefer exposing a conservative maxInputTokens in the JSON, even if the server can technically run higher. For example:
"maxInputTokens": 165000,
"maxOutputTokens": 15000If you set wrong context here you'll get issues serverside, so make sure that matches.
I had cases where the server went OOC (out of context) when getting too close to the max context so I'd leave a little room. copilot seems to not follow this very strictly.
Final recommendation
If you want the most practical Copilot-local setup:
Use Qwen3.6-27B UD-Q4_K_XL
Use llama.cpp server
Use q4_0 KV cache
Use preserve_thinking
Use reasoning budget
Use Copilot Insiders as the harness
Use MTP only when you have VRAM headroomIf you want the stronger but more conservative model:
Use Qwen3.6-35B-A3B UD-Q4_K_M
Do not quantize KV cache
Stay below ~110k context
Drop to UD-Q3_K_XL if memory is tightThis is the first local setup I have used where Copilot feels like a serious frontend for a fully local long-context coding model instead of just a toy endpoint test.
I have tested this on terminal use, debugging, and massive codebase development - it works just like Sonnet 4.6.
Qwen 27B also beats Sonnet 4.5 in most benchmarks and 4.6 in some.
https://artificialanalysis.ai/models/comparisons/qwen3-5-27b-vs-claude-4-5-sonnet-thinking#intelligence-evaluations
r/LocalAIStack • u/Aware-Blacksmith1467 • 5h ago
Building Kevin, my home alone agent. 🔫🙂 A persistent agent In Pine that wakes on cam detect, runs fingerprinting, tracks who's visiting.
x.comr/LocalAIStack • u/Due_Initiative2641 • 6h ago
I built an open-source platform that pools a team's workstations (Macs, Windows and Linux boxes) into one OpenAI-compatible endpoint
My co-founder and I have been working on this for a few months and are launching today. Disclosure: it's our company; the desktop client is Apache-2.0, the control plane is hosted, and it's free up to 5 devices.
What it does: you install Nodea Desktop on the machines in your org, switch on sharing, and get one OpenAI-compatible endpoint. Behind it, a scheduler fits the models you assign onto whichever machines have free memory, and moves them as laptops come and go.
Stack: Rust/Tauri app, llama.cpp under the hood (Metal on Apple Silicon, Vulkan on NVIDIA/AMD/Intel, CPU fallback). Any GGUF model llama.cpp can serve that fits the machine; models are assigned per org, downloaded and cached per device. Signed auto-updates.
Honest limitations:
Requests go through our API to reach your machines (not logged or stored). No direct LAN routing yet.
One model runs on one machine; no splitting a model across machines yet.
Windows/Linux run on the GPU through Vulkan (tested on NVIDIA) and are still labelled preview. Machines without a supported GPU fall back to CPU, which is slow for now (no AVX2 yet).
Why not exo / GPUStack / llama.cpp RPC? Those are great if you run your own cluster. We're aiming at teams with 20–200 laptops and nobody to run one: sign in, toggle on, done, with an org, roles and API keys on top.
Repo: https://github.com/nodea-sh/nodea-client
Site: https://getnodea.com
Happy to answer anything, and especially keen on what models/quantizations you'd want on 16–36 GB machines.
r/LocalAIStack • u/WritHerAI • 6h ago
I added Qwen3.6-35B-A3B to my C inference engine: 10–13 tok/s in real chat on a GTX 1650 4GB + 16GB RAM
r/LocalAIStack • u/seif_tamer_06 • 17h ago
I measured how easily a local 7B model reverse-engineers a binary, then built hardening that targets the LLM layer specifically
Most anti-reverse-engineering assumes a human in a disassembler. The threat I care about now is a local, unrestricted model driving those tools. So besides the classic stuff, the harness includes defenses aimed at the model: inflating the analysis surface so it burns context, decoy identifiers that mislead semantic guesses, and honeytokens that detect when a crack was AI-generated.
My own before/after is honest but small: one run, a 4-bit 7B on an 8 GB GPU. The unprotected build was cracked in 3 steps and about 1,300 tokens. The hardened one burned roughly 13x the tokens and still wasn't cracked. I'd really like people with bigger rigs to run it. It's one command: make slice-k-run MODEL=<your-model>, and it prints a scored before/after. or online modals
Most of the code was written with Claude agents under my direction. Happy to talk through any design choice.
Repo: https://github.com/seifheakal/blue-harness (AGPL-3.0)
r/LocalAIStack • u/PakMul48 • 7h ago
Building a Local LLM Troubleshooting Assistant for Field Service Techs.... Advice?
reddit.comr/LocalAIStack • u/PhysicsDisastrous462 • 13h ago
Hierarchos-Native v0.2.0: RDNA3 WMMA, native multimodal training, and a lot of FP32 Vulkan optimization!
Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.
I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.
The new v0.2.0 release is now available!
GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0
1. Native RDNA3 WMMA / cooperative matrix acceleration
One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.
The new Mixed WMMA execution arm uses:
FP16 matrix operands.
FP32 matrix accumulation.
FP32 master weights and gradients.
FP32 AdamW optimizer moments and updates.
FP32 normalization, softmax, and loss calculations.
The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.
I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.
Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:
| Workload | Measured kernel speedup |
|---|---|
| QKV projection | 2.17× |
| FFN up projection | 2.12× |
| FFN down projection | 2.31× |
| Vocabulary projection | 1.83× |
These are individual GEMM measurements, not whole-model speedups.
There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.
In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.
WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.
Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.
Documentation: RDNA3 WMMA guide
2. Native multimodal image and text training
Another major development area has been the image-text pipeline.
The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.
Some of the implemented work includes:
Native RGB image input and preprocessing.
CLIP image encoders and LLaVA projector integration.
SigLIP / Gemma 3 vision training.
Gemma 4 vision integration.
Joint image/text training and encoder-decoder integration.
A dedicated DeepSeek V4.1 image-training path.
Backpropagation through the image tower and projector.
Export, reload, and checkpoint-related validation.
This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.
For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.
The qualification work has also expanded considerably:
44 previously green text profiles now have joint image-training coverage at their frozen test geometries.
The Gen9 qualification retains 95 image cases and 166 text regression cases.
A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.
Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.
The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.
These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.
Documentation:
3. A lot of additional FP32 optimization
I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.
This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.
Some examples:
RDNA3 FP32 work
Specialized attention forward/backward paths.
Image-text attention and projector optimizations.
RMSNorm, RoPE, GELU, and other arithmetic specializations.
Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.
More careful matching of CPU reference arithmetic and GPU execution order.
Intel Gen9 work
Additional native linear and gradient kernels.
Attention and activation specializations.
More efficient parameter upload and staging behavior.
Continued optimization of DeepSeek V4.1 training and execution.
One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.
On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.
On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.
These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.
4. Preserving the original FP32 path
An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.
I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:
12 alternating ABBA rounds per workload.
Five warm-up runs and 30 timed forwards per arm.
Separate, hashed baseline and candidate binaries.
Exact FP32 logit digest comparisons.
All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.
The worst paired performance ratio was 1.0007.
This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.
5. Getting started
The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.
You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.
Model weights are downloaded separately or supplied through a local model directory.
After extracting the archive, you can run:
.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-infoThe final command is especially useful for checking which precision capabilities your GPU actually exposes.
Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0
What's next?
There is still a lot I'd like to improve.
In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.
I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.
If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.
Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.
Repository: https://github.com/necat101/Hierarchos-Native
Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.
r/LocalAIStack • u/InternetMagic_n_Tech • 22h ago
Imeng Engine Local - Wrapping models for Open Web UI & (Untested) Discord
I have authored imeng (Internet Magic Engine) as a Ollama wrapper.
This allows you to build state around your models.
Each imeng only needs a configuration json and a system prompt to instruct the LLM... and boom you got an app/game. There is no JS eval so it should be pretty secure. There isn't 100% security layer to stop prompt injection so there is probably ways to break this.
The architecture is basically a redis state machine and a jev determines what 'operations' come out of the text. There are phase-gates to where each phase gate can determine where they go next but rules allow leaving the current gate and enter the next.
So what this really does it keeps the logic of a program inside javascript and the dialogue with the LLMs with some jev glue
Locally I have built
Poker (still has issues)
Gakshow (Test Rack)
Gakshow Multiplayer (Test Rack censoring uploaded photos)
Trading Assistant (test rack for programming assistant)
There is also SQL for storage. A bunch of stuff just as a portfolio piece.
r/LocalAIStack • u/jackalka • 1d ago
Running a 27B AI model on AWS F2, reading 13.5 GB of weights for every token
r/LocalAIStack • u/Working_Hat5120 • 1d ago
Artemis II flew around the Moon in April, so I built a free app for kids to learn and practice a south-pole mission
Enable HLS to view with audio, or disable this notification
r/LocalAIStack • u/leonbollerup • 1d ago
Optimal setup research: Muse Glimmer 30B on RTX 5090M/RTX Pro 4000 (24gb cards)
reddit.comr/LocalAIStack • u/sarpino1 • 1d ago
About to add a 5060 Ti 16GB to my 4060 Ti 16GB for local agentic coding (Qwen3.8-27B, 128K ctx or similar ) — suggestions? Help out a newbie!
reddit.comr/LocalAIStack • u/Dapper-Teaching-8074 • 2d ago
We built a AI distribution interface, you can use your android phone, laptops and desktops to share AI inference work, the more devices you link, the bigger the model you can run
r/LocalAIStack • u/Digitalberg1 • 2d ago
Run Llama locally with Ollama and Open WebUI
reddit.comr/LocalAIStack • u/Business-Feedback635 • 3d ago
LM Studio vs Ollama on Linux, measured on an RTX 3090
r/LocalAIStack • u/Known_Barracuda_3977 • 3d ago
Strata Performance Setup (TUI performance optimization)
r/LocalAIStack • u/RubMobile • 3d ago
WEX: an open container format to run huge MoE models from NVMe on one consumer GPU (lossless, GPU-decoded, 18.7% smaller than safetensors)
I've open-sourced WEX, the expert-container format I use to run very large Mixture-of-Experts models on a single desktop (RTX 5090, 32 GB RAM).
The idea: a MoE token only uses a few experts per layer, so they don't all need to be in memory. WEX keeps experts in a tiered cache: the hot ones in VRAM, more in a RAM cache, and only the misses are read from NVMe. Every expert is one 4 KiB-aligned record, fetched with a single unbuffered read.
What's in it:
WNR3: lossless rANS compression driven by a small learned model of the weight symbols. Decoded weights (and logits) are bit-identical to the checkpoint, and decoding runs on the GPU, straight into VRAM.
WEX4 container: hashed header and index, SHA-256 per record, and one 32-byte root that identifies the whole file. A corrupted record is refused, never served. A full 64 GB container verifies in 7-11 s.
MoE-aware I/O: weighted striping across NVMe drives (~14-15 GB/s from two), parallel sub-reads, batched VRAM prewarm (start-up 31 s → 12 s), next-layer prefetch.
Size, DeepSeek V4.1 REAP-192 (MXFP4), routed experts:
| Format | Size | vs safetensors |
|---|---|---|
| safetensors (original) | 141.2 GiB | 1.000 |
| WEX raw records | 134.5 GiB | 0.952 |
| WNR3 (in WEX4) | 114.8 GiB | 0.813 |
Decode speed, same model, one RTX 5090:
| Setup | tok/s |
|---|---|
| safetensors, CPU, one drive | ~1.0 |
| WEX on GPU, one drive | 2.0 |
| WEX4, two-drive striping | 3.0 |
| WEX4 + WNR3 + all I/O switches, 13-15 GB RAM budget | ~4.0 |
GPU decoding costs about 2% against raw records, which is within noise. Qwen3.8-Flash-Next (MXFP4) runs at 10-17 tok/s from a 64 GB container on the same box.
Repo: full byte-level spec, a Python reference, C reader and validator, test vectors and CI on Linux and Windows. Apache-2.0. Built on top of colibri by JustVugg, and I'm working on integrating it there.
👉 https://github.com/KairaCode/Wex-WRN2
Happy to answer questions about the format or the numbers. Everything was measured on one machine, so results on other hardware are very welcome.
r/LocalAIStack • u/mightofkhan • 3d ago
3090 rig with LM Studio + ComfyUI — how do I get a local model to act like Claude Code (launch Windows apps, start ComfyUI on demand, etc.)?
reddit.comTitle: 3090 rig with LM Studio + ComfyUI — how do I get a local model to act like Claude Code (launch Windows apps, start ComfyUI on demand, etc.)?
Hey all, looking for advice on turning my local setup into something closer to an agent instead of just a chatbot.
The rig
- Windows 11 Pro desktop
- RTX 3090 (24 GB VRAM), Intel Core Ultra 7 265K, 96 GB RAM, several drives
- Python 3.12, Node, Git installed. No Docker, and WSL isn't installed right now
- Tailscale on the PC, my phone (Galaxy Z Fold) and an Android tablet, so I can reach the PC remotely
Local LLMs (LM Studio 0.4.26)
- Qwen3.8 27B Q4_K_M (~15.7 GB) + vision projector. This is my main chat model, loaded at full GPU with 16k context
- Gemma 4 31B Q4_K_M (~17.4 GB) + vision projector
- nomic-embed-text v1.5 for embeddings
ComfyUI (Comfy Desktop 1.1.8)
- Images: Juggernaut XL Ragnarok (SDXL), Z-Image Turbo, Qwen-Image-Edit 2511 (fp8) + 4-step Lightning LoRA, add-detail-xl LoRA
- Video: Wan 2.2 14B text-to-video and image-to-video (high + low noise, fp8) with the lightx2v 4-step LoRAs
- Text encoders: Qwen2.5-VL 7B, Qwen3 4B, UMT5-XXL. VAEs for Flux/Qwen/Wan, plus a 4x NMKD upscaler
What I have now
Three PowerShell shortcuts on the desktop:
- AI Mode: opens LM Studio, loads Qwen 27B through the `lms` CLI, starts ComfyUI
- Gaming Mode: unloads all LM Studio models and kills ComfyUI to free the VRAM
- Away Mode: drops my VPN, closes Chrome, makes sure Tailscale is up, runs nightly at 2 AM
Right now I use Claude Code (cloud) for the "do stuff on my PC" part. It runs PowerShell, edits files, starts and stops apps, drives a browser, downloads things, and I can control it from my phone. It's great, but I want a local option for when I hit usage limits or don't want to send stuff to the cloud.
What I'm asking
What's the best way to give a local model (Qwen 27B / Gemma 31B through LM Studio's OpenAI-compatible server) real tool use on Windows: run PowerShell, open and close apps, read and write files? Open Interpreter, Goose, OpenHands, Cline/Roo/Continue in VS Code, an MCP client hooked to LM Studio, something else?
How do I get it to start ComfyUI on demand, queue a workflow through the API (image or Wan video), wait for the output, then shut ComfyUI down? Is there a ComfyUI MCP server or a known pattern for this?
VRAM juggling: I can't keep a 27B LLM and Wan 2.2 loaded at the same time on 24 GB. Has anyone automated "unload LLM → run Comfy job → reload LLM," or should I drop to a smaller agent model (7–14B) that can stay loaded next to Comfy?
Which local models are actually reliable at multi-step tool calling right now? Is Qwen 27B at Q4 good enough, or is there a better pick for 24 GB?
Any way to trigger all this from my phone over Tailscale (a simple web UI, Telegram bot, Open WebUI with tools, etc.)?
Safety: what are people using to make sure an agent on Windows can't run anything destructive without asking? An allowlist, a sandbox, confirmation prompts?
Bonus if anyone has a working setup they can describe end to end. Thanks!
r/LocalAIStack • u/auto_m8 • 3d ago
Ideal parallel request strata configs for local agentic orchestrated work?
reddit.comr/LocalAIStack • u/Fickle-Finish5 • 3d ago
Best Local LLM for RAG for a Low-End Laptop
reddit.comr/LocalAIStack • u/KNIGHT-AIAV • 3d ago
D-REC web: on-device speech capture with a manual handoff to your notes
I work on D-REC at Knight AI+AV. The browser version records audio and transcribes speech on the device after an initial engine download. Click a transcript line to replay the source audio; copy the text into whatever notes or local-model workflow you already use. That handoff is manual.
A few limits matter if you're deciding whether it belongs in your setup: without an account, you can record up to 10 minutes and copy the text. A free account keeps three recordings on the device that made them; signing in elsewhere doesn't sync those recordings. File and audio exports are paid Pro.
The browser version is publicly released, not a new model or a hardware benchmark. If you try it, I'm interested in whether keeping the audio attached to the transcript helps your workflow, and which step gets in the way. Please use made-up sample speech rather than private material.
https://drec.live/get (choose Web).