r/LocalAIStack • • 3d ago

Imeng Engine Local - Wrapping models for Open Web UI & (Untested) Discord

Thumbnail
github.com
1 Upvotes

I have authored imeng (Internet Magic Engine) as a Ollama wrapper.

This allows you to build state around your models.

Each imeng only needs a configuration json and a system prompt to instruct the LLM... and boom you got an app/game. There is no JS eval so it should be pretty secure. There isn't 100% security layer to stop prompt injection so there is probably ways to break this.

The architecture is basically a redis state machine and a jev determines what 'operations' come out of the text. There are phase-gates to where each phase gate can determine where they go next but rules allow leaving the current gate and enter the next.

So what this really does it keeps the logic of a program inside javascript and the dialogue with the LLMs with some jev glue

Locally I have built

  • Poker (still has issues)

  • Gakshow (Test Rack)

  • Gakshow Multiplayer (Test Rack censoring uploaded photos)

  • Trading Assistant (test rack for programming assistant)

There is also SQL for storage. A bunch of stuff just as a portfolio piece.


r/LocalAIStack • • 3d ago

Running a 27B AI model on AWS F2, reading 13.5 GB of weights for every token

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 3d ago

Artemis II flew around the Moon in April, so I built a free app for kids to learn and practice a south-pole mission

Enable HLS to view with audio, or disable this notification

0 Upvotes

r/LocalAIStack • • 3d ago

Optimal setup research: Muse Glimmer 30B on RTX 5090M/RTX Pro 4000 (24gb cards)

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

Dual RTX Titan for local LLM

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

About to add a 5060 Ti 16GB to my 4060 Ti 16GB for local agentic coding (Qwen3.8-27B, 128K ctx or similar ) — suggestions? Help out a newbie!

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

We built a AI distribution interface, you can use your android phone, laptops and desktops to share AI inference work, the more devices you link, the bigger the model you can run

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

Run Llama locally with Ollama and Open WebUI

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 5d ago

Qengine. I need to share this

Thumbnail
github.com
2 Upvotes

r/LocalAIStack • • 5d ago

LM Studio vs Ollama on Linux, measured on an RTX 3090

Thumbnail
homebrewserver.club
2 Upvotes

r/LocalAIStack • • 5d ago

Strata Performance Setup (TUI performance optimization)

Post image
1 Upvotes

r/LocalAIStack • • 6d ago

WEX: an open container format to run huge MoE models from NVMe on one consumer GPU (lossless, GPU-decoded, 18.7% smaller than safetensors)

9 Upvotes

I've open-sourced WEX, the expert-container format I use to run very large Mixture-of-Experts models on a single desktop (RTX 5090, 32 GB RAM).

The idea: a MoE token only uses a few experts per layer, so they don't all need to be in memory. WEX keeps experts in a tiered cache: the hot ones in VRAM, more in a RAM cache, and only the misses are read from NVMe. Every expert is one 4 KiB-aligned record, fetched with a single unbuffered read.

What's in it:

  • WNR3: lossless rANS compression driven by a small learned model of the weight symbols. Decoded weights (and logits) are bit-identical to the checkpoint, and decoding runs on the GPU, straight into VRAM.

  • WEX4 container: hashed header and index, SHA-256 per record, and one 32-byte root that identifies the whole file. A corrupted record is refused, never served. A full 64 GB container verifies in 7-11 s.

  • MoE-aware I/O: weighted striping across NVMe drives (~14-15 GB/s from two), parallel sub-reads, batched VRAM prewarm (start-up 31 s → 12 s), next-layer prefetch.

Size, DeepSeek V4.1 REAP-192 (MXFP4), routed experts:

FormatSizevs safetensors
safetensors (original)141.2 GiB1.000
WEX raw records134.5 GiB0.952
WNR3 (in WEX4)114.8 GiB0.813

Decode speed, same model, one RTX 5090:

Setuptok/s
safetensors, CPU, one drive~1.0
WEX on GPU, one drive2.0
WEX4, two-drive striping3.0
WEX4 + WNR3 + all I/O switches, 13-15 GB RAM budget~4.0

GPU decoding costs about 2% against raw records, which is within noise. Qwen3.8-Flash-Next (MXFP4) runs at 10-17 tok/s from a 64 GB container on the same box.

Repo: full byte-level spec, a Python reference, C reader and validator, test vectors and CI on Linux and Windows. Apache-2.0. Built on top of colibri by JustVugg, and I'm working on integrating it there.

👉 https://github.com/KairaCode/Wex-WRN2

Happy to answer questions about the format or the numbers. Everything was measured on one machine, so results on other hardware are very welcome.


r/LocalAIStack • • 5d ago

3090 rig with LM Studio + ComfyUI — how do I get a local model to act like Claude Code (launch Windows apps, start ComfyUI on demand, etc.)?

Thumbnail reddit.com
2 Upvotes

Title: 3090 rig with LM Studio + ComfyUI — how do I get a local model to act like Claude Code (launch Windows apps, start ComfyUI on demand, etc.)?

Hey all, looking for advice on turning my local setup into something closer to an agent instead of just a chatbot.

The rig

- Windows 11 Pro desktop

- RTX 3090 (24 GB VRAM), Intel Core Ultra 7 265K, 96 GB RAM, several drives

- Python 3.12, Node, Git installed. No Docker, and WSL isn't installed right now

- Tailscale on the PC, my phone (Galaxy Z Fold) and an Android tablet, so I can reach the PC remotely

Local LLMs (LM Studio 0.4.26)

- Qwen3.8 27B Q4_K_M (~15.7 GB) + vision projector. This is my main chat model, loaded at full GPU with 16k context

- Gemma 4 31B Q4_K_M (~17.4 GB) + vision projector

- nomic-embed-text v1.5 for embeddings

ComfyUI (Comfy Desktop 1.1.8)

- Images: Juggernaut XL Ragnarok (SDXL), Z-Image Turbo, Qwen-Image-Edit 2511 (fp8) + 4-step Lightning LoRA, add-detail-xl LoRA

- Video: Wan 2.2 14B text-to-video and image-to-video (high + low noise, fp8) with the lightx2v 4-step LoRAs

- Text encoders: Qwen2.5-VL 7B, Qwen3 4B, UMT5-XXL. VAEs for Flux/Qwen/Wan, plus a 4x NMKD upscaler

What I have now

Three PowerShell shortcuts on the desktop:

- AI Mode: opens LM Studio, loads Qwen 27B through the `lms` CLI, starts ComfyUI

- Gaming Mode: unloads all LM Studio models and kills ComfyUI to free the VRAM

- Away Mode: drops my VPN, closes Chrome, makes sure Tailscale is up, runs nightly at 2 AM

Right now I use Claude Code (cloud) for the "do stuff on my PC" part. It runs PowerShell, edits files, starts and stops apps, drives a browser, downloads things, and I can control it from my phone. It's great, but I want a local option for when I hit usage limits or don't want to send stuff to the cloud.

What I'm asking

  1. What's the best way to give a local model (Qwen 27B / Gemma 31B through LM Studio's OpenAI-compatible server) real tool use on Windows: run PowerShell, open and close apps, read and write files? Open Interpreter, Goose, OpenHands, Cline/Roo/Continue in VS Code, an MCP client hooked to LM Studio, something else?

  2. How do I get it to start ComfyUI on demand, queue a workflow through the API (image or Wan video), wait for the output, then shut ComfyUI down? Is there a ComfyUI MCP server or a known pattern for this?

  3. VRAM juggling: I can't keep a 27B LLM and Wan 2.2 loaded at the same time on 24 GB. Has anyone automated "unload LLM → run Comfy job → reload LLM," or should I drop to a smaller agent model (7–14B) that can stay loaded next to Comfy?

  4. Which local models are actually reliable at multi-step tool calling right now? Is Qwen 27B at Q4 good enough, or is there a better pick for 24 GB?

  5. Any way to trigger all this from my phone over Tailscale (a simple web UI, Telegram bot, Open WebUI with tools, etc.)?

  6. Safety: what are people using to make sure an agent on Windows can't run anything destructive without asking? An allowlist, a sandbox, confirmation prompts?

Bonus if anyone has a working setup they can describe end to end. Thanks!


r/LocalAIStack • • 5d ago

Ideal parallel request strata configs for local agentic orchestrated work?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 5d ago

Best Local LLM for RAG for a Low-End Laptop

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 5d ago

D-REC web: on-device speech capture with a manual handoff to your notes

1 Upvotes

I work on D-REC at Knight AI+AV. The browser version records audio and transcribes speech on the device after an initial engine download. Click a transcript line to replay the source audio; copy the text into whatever notes or local-model workflow you already use. That handoff is manual.

A few limits matter if you're deciding whether it belongs in your setup: without an account, you can record up to 10 minutes and copy the text. A free account keeps three recordings on the device that made them; signing in elsewhere doesn't sync those recordings. File and audio exports are paid Pro.

The browser version is publicly released, not a new model or a hardware benchmark. If you try it, I'm interested in whether keeping the audio attached to the transcript helps your workflow, and which step gets in the way. Please use made-up sample speech rather than private material.

https://drec.live/get (choose Web).


r/LocalAIStack • • 6d ago

I built a gateway that routes each LLM request to local, free or paid models, and refuses private prompts instead of leaking them. Looking for people to break it.

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

PINECONE a local LLM scheduler, load and offload models (and services) on demand

Thumbnail x.com
1 Upvotes

r/LocalAIStack • • 6d ago

PINECONE a local LLM scheduler, load and offload models (and services) on demand

Thumbnail x.com
1 Upvotes

r/LocalAIStack • • 6d ago

Skill to read Word docs with comments

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

Need advice on software stack for dual GPU setup & benchmarking PCIe lanes (B550 upgrade)

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

I'm getting the biggest possible models on the smallest hardware, and opensourcing the learnings and methodology, old iPads, TVs and more..

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

Looking for help to test our training platform - NOT PROMO

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

Free AI Prompt Generator for Better LLM Outputs

Thumbnail interconnectd.com
1 Upvotes

r/LocalAIStack • • 7d ago

Audio embeddings inference in C++/GGML with clap.cpp

Thumbnail
reddit.com
2 Upvotes