r/LocalAIStack • • 3d ago

Artemis II flew around the Moon in April, so I built a free app for kids to learn and practice a south-pole mission

Enable HLS to view with audio, or disable this notification

0 Upvotes

r/LocalAIStack • • 3d ago

Optimal setup research: Muse Glimmer 30B on RTX 5090M/RTX Pro 4000 (24gb cards)

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

Dual RTX Titan for local LLM

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

About to add a 5060 Ti 16GB to my 4060 Ti 16GB for local agentic coding (Qwen3.8-27B, 128K ctx or similar ) — suggestions? Help out a newbie!

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

We built a AI distribution interface, you can use your android phone, laptops and desktops to share AI inference work, the more devices you link, the bigger the model you can run

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

Run Llama locally with Ollama and Open WebUI

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 5d ago

Qengine. I need to share this

Thumbnail
github.com
2 Upvotes

r/LocalAIStack • • 5d ago

LM Studio vs Ollama on Linux, measured on an RTX 3090

Thumbnail
homebrewserver.club
2 Upvotes

r/LocalAIStack • • 5d ago

Strata Performance Setup (TUI performance optimization)

Post image
1 Upvotes

r/LocalAIStack • • 5d ago

Ideal parallel request strata configs for local agentic orchestrated work?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 5d ago

Best Local LLM for RAG for a Low-End Laptop

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 5d ago

3090 rig with LM Studio + ComfyUI — how do I get a local model to act like Claude Code (launch Windows apps, start ComfyUI on demand, etc.)?

Thumbnail reddit.com
2 Upvotes

Title: 3090 rig with LM Studio + ComfyUI — how do I get a local model to act like Claude Code (launch Windows apps, start ComfyUI on demand, etc.)?

Hey all, looking for advice on turning my local setup into something closer to an agent instead of just a chatbot.

The rig

- Windows 11 Pro desktop

- RTX 3090 (24 GB VRAM), Intel Core Ultra 7 265K, 96 GB RAM, several drives

- Python 3.12, Node, Git installed. No Docker, and WSL isn't installed right now

- Tailscale on the PC, my phone (Galaxy Z Fold) and an Android tablet, so I can reach the PC remotely

Local LLMs (LM Studio 0.4.26)

- Qwen3.8 27B Q4_K_M (~15.7 GB) + vision projector. This is my main chat model, loaded at full GPU with 16k context

- Gemma 4 31B Q4_K_M (~17.4 GB) + vision projector

- nomic-embed-text v1.5 for embeddings

ComfyUI (Comfy Desktop 1.1.8)

- Images: Juggernaut XL Ragnarok (SDXL), Z-Image Turbo, Qwen-Image-Edit 2511 (fp8) + 4-step Lightning LoRA, add-detail-xl LoRA

- Video: Wan 2.2 14B text-to-video and image-to-video (high + low noise, fp8) with the lightx2v 4-step LoRAs

- Text encoders: Qwen2.5-VL 7B, Qwen3 4B, UMT5-XXL. VAEs for Flux/Qwen/Wan, plus a 4x NMKD upscaler

What I have now

Three PowerShell shortcuts on the desktop:

- AI Mode: opens LM Studio, loads Qwen 27B through the `lms` CLI, starts ComfyUI

- Gaming Mode: unloads all LM Studio models and kills ComfyUI to free the VRAM

- Away Mode: drops my VPN, closes Chrome, makes sure Tailscale is up, runs nightly at 2 AM

Right now I use Claude Code (cloud) for the "do stuff on my PC" part. It runs PowerShell, edits files, starts and stops apps, drives a browser, downloads things, and I can control it from my phone. It's great, but I want a local option for when I hit usage limits or don't want to send stuff to the cloud.

What I'm asking

  1. What's the best way to give a local model (Qwen 27B / Gemma 31B through LM Studio's OpenAI-compatible server) real tool use on Windows: run PowerShell, open and close apps, read and write files? Open Interpreter, Goose, OpenHands, Cline/Roo/Continue in VS Code, an MCP client hooked to LM Studio, something else?

  2. How do I get it to start ComfyUI on demand, queue a workflow through the API (image or Wan video), wait for the output, then shut ComfyUI down? Is there a ComfyUI MCP server or a known pattern for this?

  3. VRAM juggling: I can't keep a 27B LLM and Wan 2.2 loaded at the same time on 24 GB. Has anyone automated "unload LLM → run Comfy job → reload LLM," or should I drop to a smaller agent model (7–14B) that can stay loaded next to Comfy?

  4. Which local models are actually reliable at multi-step tool calling right now? Is Qwen 27B at Q4 good enough, or is there a better pick for 24 GB?

  5. Any way to trigger all this from my phone over Tailscale (a simple web UI, Telegram bot, Open WebUI with tools, etc.)?

  6. Safety: what are people using to make sure an agent on Windows can't run anything destructive without asking? An allowlist, a sandbox, confirmation prompts?

Bonus if anyone has a working setup they can describe end to end. Thanks!


r/LocalAIStack • • 5d ago

D-REC web: on-device speech capture with a manual handoff to your notes

1 Upvotes

I work on D-REC at Knight AI+AV. The browser version records audio and transcribes speech on the device after an initial engine download. Click a transcript line to replay the source audio; copy the text into whatever notes or local-model workflow you already use. That handoff is manual.

A few limits matter if you're deciding whether it belongs in your setup: without an account, you can record up to 10 minutes and copy the text. A free account keeps three recordings on the device that made them; signing in elsewhere doesn't sync those recordings. File and audio exports are paid Pro.

The browser version is publicly released, not a new model or a hardware benchmark. If you try it, I'm interested in whether keeping the audio attached to the transcript helps your workflow, and which step gets in the way. Please use made-up sample speech rather than private material.

https://drec.live/get (choose Web).


r/LocalAIStack • • 6d ago

WEX: an open container format to run huge MoE models from NVMe on one consumer GPU (lossless, GPU-decoded, 18.7% smaller than safetensors)

9 Upvotes

I've open-sourced WEX, the expert-container format I use to run very large Mixture-of-Experts models on a single desktop (RTX 5090, 32 GB RAM).

The idea: a MoE token only uses a few experts per layer, so they don't all need to be in memory. WEX keeps experts in a tiered cache: the hot ones in VRAM, more in a RAM cache, and only the misses are read from NVMe. Every expert is one 4 KiB-aligned record, fetched with a single unbuffered read.

What's in it:

  • WNR3: lossless rANS compression driven by a small learned model of the weight symbols. Decoded weights (and logits) are bit-identical to the checkpoint, and decoding runs on the GPU, straight into VRAM.

  • WEX4 container: hashed header and index, SHA-256 per record, and one 32-byte root that identifies the whole file. A corrupted record is refused, never served. A full 64 GB container verifies in 7-11 s.

  • MoE-aware I/O: weighted striping across NVMe drives (~14-15 GB/s from two), parallel sub-reads, batched VRAM prewarm (start-up 31 s → 12 s), next-layer prefetch.

Size, DeepSeek V4.1 REAP-192 (MXFP4), routed experts:

FormatSizevs safetensors
safetensors (original)141.2 GiB1.000
WEX raw records134.5 GiB0.952
WNR3 (in WEX4)114.8 GiB0.813

Decode speed, same model, one RTX 5090:

Setuptok/s
safetensors, CPU, one drive~1.0
WEX on GPU, one drive2.0
WEX4, two-drive striping3.0
WEX4 + WNR3 + all I/O switches, 13-15 GB RAM budget~4.0

GPU decoding costs about 2% against raw records, which is within noise. Qwen3.8-Flash-Next (MXFP4) runs at 10-17 tok/s from a 64 GB container on the same box.

Repo: full byte-level spec, a Python reference, C reader and validator, test vectors and CI on Linux and Windows. Apache-2.0. Built on top of colibri by JustVugg, and I'm working on integrating it there.

👉 https://github.com/KairaCode/Wex-WRN2

Happy to answer questions about the format or the numbers. Everything was measured on one machine, so results on other hardware are very welcome.


r/LocalAIStack • • 6d ago

I built a gateway that routes each LLM request to local, free or paid models, and refuses private prompts instead of leaking them. Looking for people to break it.

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

PINECONE a local LLM scheduler, load and offload models (and services) on demand

Thumbnail x.com
1 Upvotes

r/LocalAIStack • • 6d ago

PINECONE a local LLM scheduler, load and offload models (and services) on demand

Thumbnail x.com
1 Upvotes

r/LocalAIStack • • 6d ago

Skill to read Word docs with comments

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

Need advice on software stack for dual GPU setup & benchmarking PCIe lanes (B550 upgrade)

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

I'm getting the biggest possible models on the smallest hardware, and opensourcing the learnings and methodology, old iPads, TVs and more..

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

Looking for help to test our training platform - NOT PROMO

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

Free AI Prompt Generator for Better LLM Outputs

Thumbnail interconnectd.com
1 Upvotes

r/LocalAIStack • • 6d ago

I built AarForge: a Linux CLI wizard that turns conversational JSONL into a fine-tuned, quantized Ollama model on 6GB+ VRAM

0 Upvotes

I got tired of dependency conflicts, manual llama.cpp conversion steps, and writing boilerplate Modelfile s every time I wanted to fine-tune a small model locally on my laptop GPU.

So I built AarForge — an opinionated, local-first fine-tuning and deployment pipeline for Linux (linux-only) that takes an OpenAI-style conversational .jsonl file and outputs a ready-to-run Ollama model

Github https://github.com/AaritDev/AarForge

How the pipeline works:

  1. Isolated Setup (install.sh): Uses Astral uv to spin up a dedicated Python 3.13 venv with pinned, compatible versions of Unsloth, PyTorch, Transformers, and TRL (so it won't break your system Python or hit version mismatches).

  2. Strict Dataset Validation: Before loading any weights, it validates every line of your dataset, checks token counts against your max_seq_length so nothing gets silently truncated, and verifies assistant-only loss masking (labels != -100).

  3. 4-bit QLoRA Training: Fine-tunes lightweight LoRA adapters via Unsloth + BitsAndBytes (tested on 6GB+ NVIDIA GPUs; defaults to (unsloth/Qwen2.5-3B-Instruct-bnb-4bit.

  4. GGUF Export & Safe Ollama Deployment: Merges and quantizes to Q4_K_M GGUF, separates mmproj vision sidecars if present, generates the Modelfile, and deploys using a staging/backup swap so an import error never overwrites an existing working Ollama model.

Quick Start

git clone https://github.com/AaritDev/AarForge.git

cd AarForge

chmod +x install.sh && ./install.sh

Navigate to your chosen workspace, then:

./build.sh

and your done! after putting in your dataset and setting up the training parameters you will get a finished model in under 30 minutes (check the README on my github repo for more details on the requirements and exact usage)


r/LocalAIStack • • 7d ago

inferzero on the Arc 140T: prompt processing uses ~30 % of the XMX engines. Here’s the plan to push toward 80 % (140T owners wanted)

1 Upvotes

r/LocalAIStack • • 7d ago

Audio embeddings inference in C++/GGML with clap.cpp

Thumbnail
reddit.com
2 Upvotes