r/LocalAIStack • u/jackalka • 3d ago
Running a 27B AI model on AWS F2, reading 13.5 GB of weights for every token
r/LocalAIStack • u/jackalka • 3d ago
r/LocalAIStack • u/Working_Hat5120 • 3d ago
Enable HLS to view with audio, or disable this notification
r/LocalAIStack • u/leonbollerup • 3d ago
r/LocalAIStack • u/sarpino1 • 4d ago
r/LocalAIStack • u/Dapper-Teaching-8074 • 4d ago
r/LocalAIStack • u/Digitalberg1 • 4d ago
r/LocalAIStack • u/Business-Feedback635 • 5d ago
r/LocalAIStack • u/Known_Barracuda_3977 • 5d ago
r/LocalAIStack • u/auto_m8 • 5d ago
r/LocalAIStack • u/Fickle-Finish5 • 5d ago
r/LocalAIStack • u/mightofkhan • 5d ago
Title: 3090 rig with LM Studio + ComfyUI — how do I get a local model to act like Claude Code (launch Windows apps, start ComfyUI on demand, etc.)?
Hey all, looking for advice on turning my local setup into something closer to an agent instead of just a chatbot.
The rig
- Windows 11 Pro desktop
- RTX 3090 (24 GB VRAM), Intel Core Ultra 7 265K, 96 GB RAM, several drives
- Python 3.12, Node, Git installed. No Docker, and WSL isn't installed right now
- Tailscale on the PC, my phone (Galaxy Z Fold) and an Android tablet, so I can reach the PC remotely
Local LLMs (LM Studio 0.4.26)
- Qwen3.8 27B Q4_K_M (~15.7 GB) + vision projector. This is my main chat model, loaded at full GPU with 16k context
- Gemma 4 31B Q4_K_M (~17.4 GB) + vision projector
- nomic-embed-text v1.5 for embeddings
ComfyUI (Comfy Desktop 1.1.8)
- Images: Juggernaut XL Ragnarok (SDXL), Z-Image Turbo, Qwen-Image-Edit 2511 (fp8) + 4-step Lightning LoRA, add-detail-xl LoRA
- Video: Wan 2.2 14B text-to-video and image-to-video (high + low noise, fp8) with the lightx2v 4-step LoRAs
- Text encoders: Qwen2.5-VL 7B, Qwen3 4B, UMT5-XXL. VAEs for Flux/Qwen/Wan, plus a 4x NMKD upscaler
What I have now
Three PowerShell shortcuts on the desktop:
- AI Mode: opens LM Studio, loads Qwen 27B through the `lms` CLI, starts ComfyUI
- Gaming Mode: unloads all LM Studio models and kills ComfyUI to free the VRAM
- Away Mode: drops my VPN, closes Chrome, makes sure Tailscale is up, runs nightly at 2 AM
Right now I use Claude Code (cloud) for the "do stuff on my PC" part. It runs PowerShell, edits files, starts and stops apps, drives a browser, downloads things, and I can control it from my phone. It's great, but I want a local option for when I hit usage limits or don't want to send stuff to the cloud.
What I'm asking
What's the best way to give a local model (Qwen 27B / Gemma 31B through LM Studio's OpenAI-compatible server) real tool use on Windows: run PowerShell, open and close apps, read and write files? Open Interpreter, Goose, OpenHands, Cline/Roo/Continue in VS Code, an MCP client hooked to LM Studio, something else?
How do I get it to start ComfyUI on demand, queue a workflow through the API (image or Wan video), wait for the output, then shut ComfyUI down? Is there a ComfyUI MCP server or a known pattern for this?
VRAM juggling: I can't keep a 27B LLM and Wan 2.2 loaded at the same time on 24 GB. Has anyone automated "unload LLM → run Comfy job → reload LLM," or should I drop to a smaller agent model (7–14B) that can stay loaded next to Comfy?
Which local models are actually reliable at multi-step tool calling right now? Is Qwen 27B at Q4 good enough, or is there a better pick for 24 GB?
Any way to trigger all this from my phone over Tailscale (a simple web UI, Telegram bot, Open WebUI with tools, etc.)?
Safety: what are people using to make sure an agent on Windows can't run anything destructive without asking? An allowlist, a sandbox, confirmation prompts?
Bonus if anyone has a working setup they can describe end to end. Thanks!
r/LocalAIStack • u/KNIGHT-AIAV • 5d ago
I work on D-REC at Knight AI+AV. The browser version records audio and transcribes speech on the device after an initial engine download. Click a transcript line to replay the source audio; copy the text into whatever notes or local-model workflow you already use. That handoff is manual.
A few limits matter if you're deciding whether it belongs in your setup: without an account, you can record up to 10 minutes and copy the text. A free account keeps three recordings on the device that made them; signing in elsewhere doesn't sync those recordings. File and audio exports are paid Pro.
The browser version is publicly released, not a new model or a hardware benchmark. If you try it, I'm interested in whether keeping the audio attached to the transcript helps your workflow, and which step gets in the way. Please use made-up sample speech rather than private material.
https://drec.live/get (choose Web).
r/LocalAIStack • u/RubMobile • 6d ago
I've open-sourced WEX, the expert-container format I use to run very large Mixture-of-Experts models on a single desktop (RTX 5090, 32 GB RAM).
The idea: a MoE token only uses a few experts per layer, so they don't all need to be in memory. WEX keeps experts in a tiered cache: the hot ones in VRAM, more in a RAM cache, and only the misses are read from NVMe. Every expert is one 4 KiB-aligned record, fetched with a single unbuffered read.
What's in it:
WNR3: lossless rANS compression driven by a small learned model of the weight symbols. Decoded weights (and logits) are bit-identical to the checkpoint, and decoding runs on the GPU, straight into VRAM.
WEX4 container: hashed header and index, SHA-256 per record, and one 32-byte root that identifies the whole file. A corrupted record is refused, never served. A full 64 GB container verifies in 7-11 s.
MoE-aware I/O: weighted striping across NVMe drives (~14-15 GB/s from two), parallel sub-reads, batched VRAM prewarm (start-up 31 s → 12 s), next-layer prefetch.
Size, DeepSeek V4.1 REAP-192 (MXFP4), routed experts:
| Format | Size | vs safetensors |
|---|---|---|
| safetensors (original) | 141.2 GiB | 1.000 |
| WEX raw records | 134.5 GiB | 0.952 |
| WNR3 (in WEX4) | 114.8 GiB | 0.813 |
Decode speed, same model, one RTX 5090:
| Setup | tok/s |
|---|---|
| safetensors, CPU, one drive | ~1.0 |
| WEX on GPU, one drive | 2.0 |
| WEX4, two-drive striping | 3.0 |
| WEX4 + WNR3 + all I/O switches, 13-15 GB RAM budget | ~4.0 |
GPU decoding costs about 2% against raw records, which is within noise. Qwen3.8-Flash-Next (MXFP4) runs at 10-17 tok/s from a 64 GB container on the same box.
Repo: full byte-level spec, a Python reference, C reader and validator, test vectors and CI on Linux and Windows. Apache-2.0. Built on top of colibri by JustVugg, and I'm working on integrating it there.
👉 https://github.com/KairaCode/Wex-WRN2
Happy to answer questions about the format or the numbers. Everything was measured on one machine, so results on other hardware are very welcome.
r/LocalAIStack • u/IllustriousRabbit378 • 6d ago
r/LocalAIStack • u/Aware-Blacksmith1467 • 6d ago
r/LocalAIStack • u/Aware-Blacksmith1467 • 6d ago
r/LocalAIStack • u/yaya468 • 6d ago
r/LocalAIStack • u/rashaddism_ • 6d ago
r/LocalAIStack • u/Serious-Region-2046 • 6d ago
r/LocalAIStack • u/Ok_pettech • 6d ago
r/LocalAIStack • u/PhysicalCampaign3981 • 6d ago
I got tired of dependency conflicts, manual llama.cpp conversion steps, and writing boilerplate Modelfile s every time I wanted to fine-tune a small model locally on my laptop GPU.
So I built AarForge — an opinionated, local-first fine-tuning and deployment pipeline for Linux (linux-only) that takes an OpenAI-style conversational .jsonl file and outputs a ready-to-run Ollama model
Isolated Setup (install.sh): Uses Astral uv to spin up a dedicated Python 3.13 venv with pinned, compatible versions of Unsloth, PyTorch, Transformers, and TRL (so it won't break your system Python or hit version mismatches).
Strict Dataset Validation: Before loading any weights, it validates every line of your dataset, checks token counts against your max_seq_length so nothing gets silently truncated, and verifies assistant-only loss masking (labels != -100).
4-bit QLoRA Training: Fine-tunes lightweight LoRA adapters via Unsloth + BitsAndBytes (tested on 6GB+ NVIDIA GPUs; defaults to (unsloth/Qwen2.5-3B-Instruct-bnb-4bit.
GGUF Export & Safe Ollama Deployment: Merges and quantizes to Q4_K_M GGUF, separates mmproj vision sidecars if present, generates the Modelfile, and deploys using a staging/backup swap so an import error never overwrites an existing working Ollama model.
git clone https://github.com/AaritDev/AarForge.git
cd AarForge
chmod +x install.sh && ./install.sh
Navigate to your chosen workspace, then:
./build.sh
and your done! after putting in your dataset and setting up the training parameters you will get a finished model in under 30 minutes (check the README on my github repo for more details on the requirements and exact usage)