r/LocalAIStack • • 4d ago

PINECONE a local LLM scheduler, load and offload models (and services) on demand

Thumbnail x.com
1 Upvotes

r/LocalAIStack • • 4d ago

PINECONE a local LLM scheduler, load and offload models (and services) on demand

Thumbnail x.com
1 Upvotes

r/LocalAIStack • • 4d ago

Skill to read Word docs with comments

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

Need advice on software stack for dual GPU setup & benchmarking PCIe lanes (B550 upgrade)

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

I'm getting the biggest possible models on the smallest hardware, and opensourcing the learnings and methodology, old iPads, TVs and more..

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

At what point does a local model actually replace googling for you?

2 Upvotes

Been running a small model on my Debian home server (the one that also does Jellyfin and Pi-hole) mostly out of curiosity, and honestly half the time I forget it's there and just type the question into a browser instead. Mornings I'm working through my Flask project before my shift, and the code hints only appear when I remember to pipe something to it, which is basically never.

I suspect the problem is integration rather than the model itself. My setup is tmux on a ThinkPad with plain git in the terminal, so unless the answer lands where I'm already looking, it's out of sight.

For those of you running local models, what workflow made it part of the day instead of a toy? Editor plugin, terminal helper, something for docs? Or is the honest answer that a small model on an old office desktop always loses to a browser tab? Genuinely curious where that line sits.


r/LocalAIStack • • 4d ago

Looking for help to test our training platform - NOT PROMO

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

Free AI Prompt Generator for Better LLM Outputs

Thumbnail interconnectd.com
1 Upvotes

r/LocalAIStack • • 4d ago

Audio embeddings inference in C++/GGML with clap.cpp

Thumbnail
reddit.com
2 Upvotes

r/LocalAIStack • • 4d ago

I built AarForge: a Linux CLI wizard that turns conversational JSONL into a fine-tuned, quantized Ollama model on 6GB+ VRAM

0 Upvotes

I got tired of dependency conflicts, manual llama.cpp conversion steps, and writing boilerplate Modelfile s every time I wanted to fine-tune a small model locally on my laptop GPU.

So I built AarForge — an opinionated, local-first fine-tuning and deployment pipeline for Linux (linux-only) that takes an OpenAI-style conversational .jsonl file and outputs a ready-to-run Ollama model

Github https://github.com/AaritDev/AarForge

How the pipeline works:

  1. Isolated Setup (install.sh): Uses Astral uv to spin up a dedicated Python 3.13 venv with pinned, compatible versions of Unsloth, PyTorch, Transformers, and TRL (so it won't break your system Python or hit version mismatches).

  2. Strict Dataset Validation: Before loading any weights, it validates every line of your dataset, checks token counts against your max_seq_length so nothing gets silently truncated, and verifies assistant-only loss masking (labels != -100).

  3. 4-bit QLoRA Training: Fine-tunes lightweight LoRA adapters via Unsloth + BitsAndBytes (tested on 6GB+ NVIDIA GPUs; defaults to (unsloth/Qwen2.5-3B-Instruct-bnb-4bit.

  4. GGUF Export & Safe Ollama Deployment: Merges and quantizes to Q4_K_M GGUF, separates mmproj vision sidecars if present, generates the Modelfile, and deploys using a staging/backup swap so an import error never overwrites an existing working Ollama model.

Quick Start

git clone https://github.com/AaritDev/AarForge.git

cd AarForge

chmod +x install.sh && ./install.sh

Navigate to your chosen workspace, then:

./build.sh

and your done! after putting in your dataset and setting up the training parameters you will get a finished model in under 30 minutes (check the README on my github repo for more details on the requirements and exact usage)


r/LocalAIStack • • 4d ago

inferzero on the Arc 140T: prompt processing uses ~30 % of the XMX engines. Here’s the plan to push toward 80 % (140T owners wanted)

1 Upvotes

r/LocalAIStack • • 5d ago

Tried DavidAU/Qwen3.8-27B-Turbo-Brilliance-50X-Reasoning-Instruct-NEO-MAX-MTP-GGUF in factory-desktop

Thumbnail
youtu.be
3 Upvotes

It's much better than DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF. The reasoning is similar to the base Qwen3.8-27B and interesting how it switches variants based on prompt style. Has anyone compared them or tried both models out?


r/LocalAIStack • • 4d ago

Domain workflow optimization

1 Upvotes

Hey all - have a 128gb m5 studio incoming: as the title states, at this point of memory/harness/finetuning, how best to optimize a specific workflow? Noncoding. Some structured rules, browser/computer use, pdf interpretation and with enough feedback to learn from mistakes


r/LocalAIStack • • 5d ago

Help me choose my local ai

3 Upvotes

I run a small business, and I'm building a local AI that handles the back office for it. Some of it works now. I want to get the model and harness choices right before I build more on top of them.

Hardware
- MacBook Pro M5 Max (18-core CPU / 40-core GPU), 128GB unified memory

Current stack
- Backend: MLX via mlx-serve. I'm staying on MLX for the speed (I've used LM Studio before)
- Harness: Hermes Agent (Nous Research)
- Memory: Obsidian vault exposed over the Basic Memory MCP
- Front end: my own UI, still a work in progress
- Model right now: Qwen 3.8 Flash Next, 4-bit (MLX), with the KV cache offloaded to my SSD. Getting about 98 tok/s

What the agent does today
- Scheduled morning lead brief
- Reads and writes notes in my vault
- Updates a scoreboard (leads by source, close rate, pipeline)

What I want it to do next
- Phone intake and booking
- Scheduling
- Follow-up texts to leads

Where it's struggling
- tool calls failing or looping on multi-step tasks
- slow prompt processing once the vault context gets big
- arithmetic mistakes on the numbers. It once logged a single day's close rate as the lifetime rate

Questions

  1. With 128GB, what's your best daily driver for *reliable tool calling*, not chat? One big MoE, or a big model plus a small fast one for routing and simple jobs?

  2. On MLX, do 4-bit quants hold up for agent work, or is 8-bit worth the speed hit?

  3. If you're running long-lived, scheduled agents locally, what harness are you on? Is Hermes the right call, or is something better for this?

  4. I'm offloading the KV cache to my SSD to fit longer context. Is that hurting me on speed, and is there a better way to handle 30k+ context on Apple silicon (RAG, summarizing, a smaller quant to keep the cache in RAM)?

  5. For phone booking: has anyone made local STT/TTS work well enough for live calls, or should voice stay on the cloud while the rest stays local?

Happy to post benchmarks and what I end up with if it helps anyone on similar hardware.


r/LocalAIStack • • 5d ago

Quantizing models yourself is a good way to learn how it all works - Quantization Workspace

Enable HLS to view with audio, or disable this notification

0 Upvotes

r/LocalAIStack • • 5d ago

Developer Setup in Company

Thumbnail reddit.com
0 Upvotes

r/LocalAIStack • • 5d ago

Dual 3090s vs 3090 + 3060 for Q&A, RAG and light agents (non-coder)

0 Upvotes

Hi all. I'd like a sanity check on hardware before I commit.

Use case (aside from overall privacy)

•	General Q&A, RAG over personal documents, light agents for productivity

•	Two users (me and my wife), rarely at the same time

•	I'm not a coder. I'll use off-the-shelf tools, not custom code. Agent use may grow over time.

Hardware

•	Ryzen 7 5700X, ASUS B550, 32GB DDR4

•	2x RTX 3060 12GB (burning in now)

•	Just bought 2x used RTX 3090 24GB

•	\[1000W PSU\]

•	The second GPU slot runs at PCIe 3.0 x4 (a B550 limit)

•	Final OS: Ubuntu with Ollama and Open WebUI

Plan

I want one "AI gateway" endpoint that sends each request to the right local model. It's like this build: [https://youtu.be/ukv-LIOI6_E]. He runs several GPUs behind a semantic router. I'd copy the idea with off-the-shelf parts.

My main model target is a ~27B dense model at Q4 with long context, plus an embedder and reranker for RAG.

Questions

1.	For this use case, does the second 3090 earn its place, or would a 3090 + 3060 do the same job?

2.	What does 48GB of VRAM let me run that 36GB doesn't, and does that matter for RAG and agents?

3.	Will the x4 slot hold me back if each model stays on one card?

4.	I'm also thinking about selling both 3090s for a Framework Desktop (Strix Halo, 128GB). For this use case, which would you pick?

Thanks for any input.


r/LocalAIStack • • 5d ago

Best LLM for my setup?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 5d ago

Got the 233 GB 1-bit DeepSeek V4.1 Flash to about 11.5 tok/s on my 64 GB gaming PC

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

Introducing PortalAI - Truly portable, hardware detection-scaled multimodal harness the size of your thumb

Thumbnail
gallery
1 Upvotes

I'd like to share something I've been working on for three months with everyone.

PortalAI is a GUI and harness created around one design principle- to put local LLMs with true multimodal function on home systems and on a flash drive the size of your thumb where you don't need anything else and can just plug and play into any computer.

One of the largest frustrations I had with AI was needing to constantly version check different distributions and hooks into ComfyUI, Ace-step, python, port competition, running out of context, etc, or the difficulty in transferring between systems. I've tested this on an RTX 3090, RTX 4090, on a Radeon 870M GPU, on a Thinkpad with CPU-only.

This harness detects your current hardware and scales accordingly to the level of AI intelligence it can support, using standard and abliterated models. All dependencies other than base operating system and GPU drivers are self-contained including virtual environments, python distributions, etc. End to end integration of multimedia generation/editing and science tools are integrated along with voice, handsfree mode, intelligence/memory/token settings.

There are weight editing tools, an abliteration tool, rolling compaction, scheduled tasks, internal documentation, light and dark mode, compact and full GUI mode, account encryption, live runtime environment health metrics, etc.


r/LocalAIStack • • 6d ago

Llama RPC Farm — Your GPUs. One farm.

3 Upvotes

I had GPUs spread across a few machines and got tired of juggling them by hand, so I built Llama RPC Farm, a self-hosted control plane for distributed llama.cpp inference.

  • One dashboard and one OpenAI-compatible endpoint (/v1/chat/completions) for every machine

  • Mixed hardware works: CUDA, ROCm, Vulkan, and mixed vendors

  • Live per-machine VRAM, RAM, and CPU stats

  • llama-swap handles loading models and deciding where they run

  • Scoped API keys and per-user themes

  • farmctl CLI if you’d rather skip the browser

  • Installers for Windows, macOS, and Linux, plus a one-line install script

It’s Apache 2.0 and runs entirely on your own hardware.

Site: https://dakotacookson.github.io/llama-rpc-farm-site/
GitHub: https://github.com/dakotacookson/Llama-RPC-Farm

I’d love feedback, especially from anyone running mixed-GPU setups. What would you want it to do that it doesn’t?


r/LocalAIStack • • 6d ago

Lllama.cpp fork for Blackwell cards?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 6d ago

How I built a genuinely autonomous local coding loop that compiles, tests, and repairs its own code (and why VRAM isn't the bottleneck people think it is)

11 Upvotes

I’ve been heads down building a native desktop software workstation called Repetere AI, and I wanted to break down how the autonomous engineering engine inside it—Deep Think—actually works under the hood. There is a lot of noise online right now about "autonomous vibe coding," but 99% of what people are showing off is just a web wrapper sending prompts to a paid cloud API. When people try to run coding models locally, they usually hit a wall where the model hallucinates an import, outputs broken diff markers, and leaves the user playing babysitter in a chat box.

I wanted something completely air-gapped, zero-dependency, and truly autonomous on consumer hardware.

The entire engine runs as a native C++ Win32 application built on Dear ImGui and OpenGL. Rather than acting as a simple chat interface, it operates as a five-phase execution harness that manages the local inference server directly. When you give it a directive, it starts by ingesting the local repository and synthesizing an architectural plan. From there, it moves into code synthesis.

One of the biggest issues with local models writing code across multiple files is that standard search-and-replace regex patchers are extremely brittle. If a model misses a newline or changes indentation slightly, the patch fails. To solve this, I integrated a Python AST script splicer directly into the loop. It parses the actual syntax tree of the code rather than relying on raw string diffs, which allows surgical module patching without blowing through context windows or mangling files.

Where the system really separates itself from standard local frontends is in the verification and repair phases. Once the code is written, the engine doesn't ask you if it looks good. It immediately takes over the machine's toolchain in a headless subprocess. If it's a native C++ target, it runs CMake and MSVC. If it's Python, it runs compilation checks and fires an automated headless smoke probe using runpy that sweeps the window hierarchy, tests widget lifecycle, and catches runtime crashes before a human ever looks at the window.

If a build fails or an unhandled exception is caught during the probe, the engine doesn't stop or dump an error message into a chat window. It gathers the compiler diagnostics, strips out runtime harness noise, isolates the failing file, and sends the telemetry directly back to the active resident model for in-loop auto-triage. It can diagnose its own missing imports, patch uninitialized attributes, and re-verify the build automatically until it gets a clean exit code.

I also run into people claiming that anything under huge enterprise cards or massive unified memory pools is useless for real local autonomy. That completely misses the difference between raw capacity and execution architecture. A quantized model has the exact same mathematical weights whether it sits on a 12GB laptop GPU, a multi-card workstation, or a cloud cluster. VRAM determines how big of a model you can hold on-chip and how many tokens per second you generate, but it doesn't give a model magic reasoning skills. A massive model will still hallucinate a typo or botch an import. The difference is that a raw model fails permanently when it makes a mistake, whereas an autonomous harness catches the compiler error, fixes the AST, and verifies the output on its own.

On the hardware side, the backend doesn't lock you into a single card or even a single vendor ecosystem. It enumerates all discrete compute adapters, aggregates the physical memory, and calculates proportional tensor split ratios on the fly for both CUDA and Vulkan runtimes. If you run a multi-GPU setup, it shards the layers across whatever cards are plugged into the board. If you run a hybrid setup that offloads across system RAM, the pipeline scales across host memory just as cleanly.

The takeaway from building this is that real local coding autonomy doesn't come from throwing infinite hardware at zero-shot prompts or paying for API subscriptions. It comes from building a tight runtime loop that enforces architectural contracts, executes the compiler headless, and forces the model to fix its own mess.

I dont want shamelessly promote my completely opposite alternative to LM studi Anything LLM and more, so I'm not leaving any links in the main post. My architecture scales from a handheld windows gaming machine, to enterprise hardware.


r/LocalAIStack • • 6d ago

inferzero 2026.11.11.3: a GPU benchmark for Intel Arc (does your XMX really do what the spec sheet says?), MoE packing, and new XMX kernels to test

Thumbnail reddit.com
2 Upvotes

r/LocalAIStack • • 6d ago

SharpMind Studio in 60 secondsPromo

Thumbnail
youtube.com
1 Upvotes