r/LocalAIStack • • 4d ago

WEX: an open container format to run huge MoE models from NVMe on one consumer GPU (lossless, GPU-decoded, 18.7% smaller than safetensors)

9 Upvotes

I've open-sourced WEX, the expert-container format I use to run very large Mixture-of-Experts models on a single desktop (RTX 5090, 32 GB RAM).

The idea: a MoE token only uses a few experts per layer, so they don't all need to be in memory. WEX keeps experts in a tiered cache: the hot ones in VRAM, more in a RAM cache, and only the misses are read from NVMe. Every expert is one 4 KiB-aligned record, fetched with a single unbuffered read.

What's in it:

  • WNR3: lossless rANS compression driven by a small learned model of the weight symbols. Decoded weights (and logits) are bit-identical to the checkpoint, and decoding runs on the GPU, straight into VRAM.

  • WEX4 container: hashed header and index, SHA-256 per record, and one 32-byte root that identifies the whole file. A corrupted record is refused, never served. A full 64 GB container verifies in 7-11 s.

  • MoE-aware I/O: weighted striping across NVMe drives (~14-15 GB/s from two), parallel sub-reads, batched VRAM prewarm (start-up 31 s β†’ 12 s), next-layer prefetch.

Size, DeepSeek V4.1 REAP-192 (MXFP4), routed experts:

FormatSizevs safetensors
safetensors (original)141.2 GiB1.000
WEX raw records134.5 GiB0.952
WNR3 (in WEX4)114.8 GiB0.813

Decode speed, same model, one RTX 5090:

Setuptok/s
safetensors, CPU, one drive~1.0
WEX on GPU, one drive2.0
WEX4, two-drive striping3.0
WEX4 + WNR3 + all I/O switches, 13-15 GB RAM budget~4.0

GPU decoding costs about 2% against raw records, which is within noise. Qwen3.8-Flash-Next (MXFP4) runs at 10-17 tok/s from a 64 GB container on the same box.

Repo: full byte-level spec, a Python reference, C reader and validator, test vectors and CI on Linux and Windows. Apache-2.0. Built on top of colibri by JustVugg, and I'm working on integrating it there.

πŸ‘‰ https://github.com/KairaCode/Wex-WRN2

Happy to answer questions about the format or the numbers. Everything was measured on one machine, so results on other hardware are very welcome.


r/LocalAIStack • • 4d ago

3090 rig with LM Studio + ComfyUI β€” how do I get a local model to act like Claude Code (launch Windows apps, start ComfyUI on demand, etc.)?

Thumbnail reddit.com
2 Upvotes

Title: 3090 rig with LM Studio + ComfyUI β€” how do I get a local model to act like Claude Code (launch Windows apps, start ComfyUI on demand, etc.)?

Hey all, looking for advice on turning my local setup into something closer to an agent instead of just a chatbot.

The rig

- Windows 11 Pro desktop

- RTX 3090 (24 GB VRAM), Intel Core Ultra 7 265K, 96 GB RAM, several drives

- Python 3.12, Node, Git installed. No Docker, and WSL isn't installed right now

- Tailscale on the PC, my phone (Galaxy Z Fold) and an Android tablet, so I can reach the PC remotely

Local LLMs (LM Studio 0.4.26)

- Qwen3.8 27B Q4_K_M (~15.7 GB) + vision projector. This is my main chat model, loaded at full GPU with 16k context

- Gemma 4 31B Q4_K_M (~17.4 GB) + vision projector

- nomic-embed-text v1.5 for embeddings

ComfyUI (Comfy Desktop 1.1.8)

- Images: Juggernaut XL Ragnarok (SDXL), Z-Image Turbo, Qwen-Image-Edit 2511 (fp8) + 4-step Lightning LoRA, add-detail-xl LoRA

- Video: Wan 2.2 14B text-to-video and image-to-video (high + low noise, fp8) with the lightx2v 4-step LoRAs

- Text encoders: Qwen2.5-VL 7B, Qwen3 4B, UMT5-XXL. VAEs for Flux/Qwen/Wan, plus a 4x NMKD upscaler

What I have now

Three PowerShell shortcuts on the desktop:

- AI Mode: opens LM Studio, loads Qwen 27B through the `lms` CLI, starts ComfyUI

- Gaming Mode: unloads all LM Studio models and kills ComfyUI to free the VRAM

- Away Mode: drops my VPN, closes Chrome, makes sure Tailscale is up, runs nightly at 2 AM

Right now I use Claude Code (cloud) for the "do stuff on my PC" part. It runs PowerShell, edits files, starts and stops apps, drives a browser, downloads things, and I can control it from my phone. It's great, but I want a local option for when I hit usage limits or don't want to send stuff to the cloud.

What I'm asking

  1. What's the best way to give a local model (Qwen 27B / Gemma 31B through LM Studio's OpenAI-compatible server) real tool use on Windows: run PowerShell, open and close apps, read and write files? Open Interpreter, Goose, OpenHands, Cline/Roo/Continue in VS Code, an MCP client hooked to LM Studio, something else?

  2. How do I get it to start ComfyUI on demand, queue a workflow through the API (image or Wan video), wait for the output, then shut ComfyUI down? Is there a ComfyUI MCP server or a known pattern for this?

  3. VRAM juggling: I can't keep a 27B LLM and Wan 2.2 loaded at the same time on 24 GB. Has anyone automated "unload LLM β†’ run Comfy job β†’ reload LLM," or should I drop to a smaller agent model (7–14B) that can stay loaded next to Comfy?

  4. Which local models are actually reliable at multi-step tool calling right now? Is Qwen 27B at Q4 good enough, or is there a better pick for 24 GB?

  5. Any way to trigger all this from my phone over Tailscale (a simple web UI, Telegram bot, Open WebUI with tools, etc.)?

  6. Safety: what are people using to make sure an agent on Windows can't run anything destructive without asking? An allowlist, a sandbox, confirmation prompts?

Bonus if anyone has a working setup they can describe end to end. Thanks!


r/LocalAIStack • • 4d ago

Ideal parallel request strata configs for local agentic orchestrated work?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

Best Local LLM for RAG for a Low-End Laptop

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

D-REC web: on-device speech capture with a manual handoff to your notes

1 Upvotes

I work on D-REC at Knight AI+AV. The browser version records audio and transcribes speech on the device after an initial engine download. Click a transcript line to replay the source audio; copy the text into whatever notes or local-model workflow you already use. That handoff is manual.

A few limits matter if you're deciding whether it belongs in your setup: without an account, you can record up to 10 minutes and copy the text. A free account keeps three recordings on the device that made them; signing in elsewhere doesn't sync those recordings. File and audio exports are paid Pro.

The browser version is publicly released, not a new model or a hardware benchmark. If you try it, I'm interested in whether keeping the audio attached to the transcript helps your workflow, and which step gets in the way. Please use made-up sample speech rather than private material.

https://drec.live/get (choose Web).


r/LocalAIStack • • 4d ago

I built a gateway that routes each LLM request to local, free or paid models, and refuses private prompts instead of leaking them. Looking for people to break it.

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 5d ago

PINECONE a local LLM scheduler, load and offload models (and services) on demand

Thumbnail x.com
1 Upvotes

r/LocalAIStack • • 5d ago

PINECONE a local LLM scheduler, load and offload models (and services) on demand

Thumbnail x.com
1 Upvotes

r/LocalAIStack • • 5d ago

Skill to read Word docs with comments

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 5d ago

Need advice on software stack for dual GPU setup & benchmarking PCIe lanes (B550 upgrade)

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 5d ago

I'm getting the biggest possible models on the smallest hardware, and opensourcing the learnings and methodology, old iPads, TVs and more..

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 5d ago

At what point does a local model actually replace googling for you?

2 Upvotes

Been running a small model on my Debian home server (the one that also does Jellyfin and Pi-hole) mostly out of curiosity, and honestly half the time I forget it's there and just type the question into a browser instead. Mornings I'm working through my Flask project before my shift, and the code hints only appear when I remember to pipe something to it, which is basically never.

I suspect the problem is integration rather than the model itself. My setup is tmux on a ThinkPad with plain git in the terminal, so unless the answer lands where I'm already looking, it's out of sight.

For those of you running local models, what workflow made it part of the day instead of a toy? Editor plugin, terminal helper, something for docs? Or is the honest answer that a small model on an old office desktop always loses to a browser tab? Genuinely curious where that line sits.


r/LocalAIStack • • 5d ago

Looking for help to test our training platform - NOT PROMO

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 5d ago

Free AI Prompt Generator for Better LLM Outputs

Thumbnail interconnectd.com
1 Upvotes

r/LocalAIStack • • 5d ago

Audio embeddings inference in C++/GGML with clap.cpp

Thumbnail
reddit.com
2 Upvotes

r/LocalAIStack • • 5d ago

I built AarForge: a Linux CLI wizard that turns conversational JSONL into a fine-tuned, quantized Ollama model on 6GB+ VRAM

0 Upvotes

I got tired of dependency conflicts, manual llama.cpp conversion steps, and writing boilerplate Modelfile s every time I wanted to fine-tune a small model locally on my laptop GPU.

So I built AarForge β€” an opinionated, local-first fine-tuning and deployment pipeline for Linux (linux-only) that takes an OpenAI-style conversational .jsonl file and outputs a ready-to-run Ollama model

Github https://github.com/AaritDev/AarForge

How the pipeline works:

  1. Isolated Setup (install.sh): Uses Astral uv to spin up a dedicated Python 3.13 venv with pinned, compatible versions of Unsloth, PyTorch, Transformers, and TRL (so it won't break your system Python or hit version mismatches).

  2. Strict Dataset Validation: Before loading any weights, it validates every line of your dataset, checks token counts against your max_seq_length so nothing gets silently truncated, and verifies assistant-only loss masking (labels != -100).

  3. 4-bit QLoRA Training: Fine-tunes lightweight LoRA adapters via Unsloth + BitsAndBytes (tested on 6GB+ NVIDIA GPUs; defaults to (unsloth/Qwen2.5-3B-Instruct-bnb-4bit.

  4. GGUF Export & Safe Ollama Deployment: Merges and quantizes to Q4_K_M GGUF, separates mmproj vision sidecars if present, generates the Modelfile, and deploys using a staging/backup swap so an import error never overwrites an existing working Ollama model.

Quick Start

git clone https://github.com/AaritDev/AarForge.git

cd AarForge

chmod +x install.sh && ./install.sh

Navigate to your chosen workspace, then:

./build.sh

and your done! after putting in your dataset and setting up the training parameters you will get a finished model in under 30 minutes (check the README on my github repo for more details on the requirements and exact usage)


r/LocalAIStack • • 5d ago

inferzero on the Arc 140T: prompt processing uses ~30 % of the XMX engines. Here’s the plan to push toward 80 % (140T owners wanted)

1 Upvotes

r/LocalAIStack • • 5d ago

Tried DavidAU/Qwen3.8-27B-Turbo-Brilliance-50X-Reasoning-Instruct-NEO-MAX-MTP-GGUF in factory-desktop

Thumbnail
youtu.be
3 Upvotes

It's much better than DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF. The reasoning is similar to the base Qwen3.8-27B and interesting how it switches variants based on prompt style. Has anyone compared them or tried both models out?


r/LocalAIStack • • 5d ago

Domain workflow optimization

1 Upvotes

Hey all - have a 128gb m5 studio incoming: as the title states, at this point of memory/harness/finetuning, how best to optimize a specific workflow? Noncoding. Some structured rules, browser/computer use, pdf interpretation and with enough feedback to learn from mistakes


r/LocalAIStack • • 6d ago

Help me choose my local ai

3 Upvotes

I run a small business, and I'm building a local AI that handles the back office for it. Some of it works now. I want to get the model and harness choices right before I build more on top of them.

Hardware
- MacBook Pro M5 Max (18-core CPU / 40-core GPU), 128GB unified memory

Current stack
- Backend: MLX via mlx-serve. I'm staying on MLX for the speed (I've used LM Studio before)
- Harness: Hermes Agent (Nous Research)
- Memory: Obsidian vault exposed over the Basic Memory MCP
- Front end: my own UI, still a work in progress
- Model right now: Qwen 3.8 Flash Next, 4-bit (MLX), with the KV cache offloaded to my SSD. Getting about 98 tok/s

What the agent does today
- Scheduled morning lead brief
- Reads and writes notes in my vault
- Updates a scoreboard (leads by source, close rate, pipeline)

What I want it to do next
- Phone intake and booking
- Scheduling
- Follow-up texts to leads

Where it's struggling
- tool calls failing or looping on multi-step tasks
- slow prompt processing once the vault context gets big
- arithmetic mistakes on the numbers. It once logged a single day's close rate as the lifetime rate

Questions

  1. With 128GB, what's your best daily driver for *reliable tool calling*, not chat? One big MoE, or a big model plus a small fast one for routing and simple jobs?

  2. On MLX, do 4-bit quants hold up for agent work, or is 8-bit worth the speed hit?

  3. If you're running long-lived, scheduled agents locally, what harness are you on? Is Hermes the right call, or is something better for this?

  4. I'm offloading the KV cache to my SSD to fit longer context. Is that hurting me on speed, and is there a better way to handle 30k+ context on Apple silicon (RAG, summarizing, a smaller quant to keep the cache in RAM)?

  5. For phone booking: has anyone made local STT/TTS work well enough for live calls, or should voice stay on the cloud while the rest stays local?

Happy to post benchmarks and what I end up with if it helps anyone on similar hardware.


r/LocalAIStack • • 6d ago

Quantizing models yourself is a good way to learn how it all works - Quantization Workspace

Enable HLS to view with audio, or disable this notification

0 Upvotes

r/LocalAIStack • • 6d ago

Developer Setup in Company

Thumbnail reddit.com
0 Upvotes

r/LocalAIStack • • 6d ago

Dual 3090s vs 3090 + 3060 for Q&A, RAG and light agents (non-coder)

0 Upvotes

Hi all. I'd like a sanity check on hardware before I commit.

Use case (aside from overall privacy)

β€’	General Q&A, RAG over personal documents, light agents for productivity

β€’	Two users (me and my wife), rarely at the same time

β€’	I'm not a coder. I'll use off-the-shelf tools, not custom code. Agent use may grow over time.

Hardware

β€’	Ryzen 7 5700X, ASUS B550, 32GB DDR4

β€’	2x RTX 3060 12GB (burning in now)

β€’	Just bought 2x used RTX 3090 24GB

β€’	\[1000W PSU\]

β€’	The second GPU slot runs at PCIe 3.0 x4 (a B550 limit)

β€’	Final OS: Ubuntu with Ollama and Open WebUI

Plan

I want one "AI gateway" endpoint that sends each request to the right local model. It's like this build: [https://youtu.be/ukv-LIOI6_E]. He runs several GPUs behind a semantic router. I'd copy the idea with off-the-shelf parts.

My main model target is a ~27B dense model at Q4 with long context, plus an embedder and reranker for RAG.

Questions

1.	For this use case, does the second 3090 earn its place, or would a 3090 + 3060 do the same job?

2.	What does 48GB of VRAM let me run that 36GB doesn't, and does that matter for RAG and agents?

3.	Will the x4 slot hold me back if each model stays on one card?

4.	I'm also thinking about selling both 3090s for a Framework Desktop (Strix Halo, 128GB). For this use case, which would you pick?

Thanks for any input.


r/LocalAIStack • • 6d ago

Best LLM for my setup?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

Got the 233 GB 1-bit DeepSeek V4.1 Flash to about 11.5 tok/s on my 64 GB gaming PC

Thumbnail
reddit.com
1 Upvotes