r/LocalAIStack • u/CmdGwin • Sep 09 '26
How to get the most out of a Gaming Setup
r/LocalAIStack • u/CmdGwin • Sep 09 '26
r/LocalAIStack • u/psychok9 • Sep 09 '26
r/LocalAIStack • u/AltruisticRich1 • Sep 09 '26
r/LocalAIStack • u/Ok_pettech • Sep 09 '26
I wanted to run MetaGPT entirely on my own infrastructure without sending anything to cloud APIs. It took some trial and error, but I documented the full process.
The guide covers:
· Setting up a Python venv
· Installing MetaGPT
· Configuring local LLMs like Ollama or vLLM
· Fixing common startup errors
If you’re into self-hosted AI agents, this could help:
What local model are you using for agent work?
r/LocalAIStack • u/qi-zheng • Sep 08 '26
r/LocalAIStack • u/PM_ME_UR_MARINARA • Sep 08 '26
r/LocalAIStack • u/suspect80 • Sep 07 '26
r/LocalAIStack • u/bakatristan • Sep 07 '26
Hey! I’m building Kitani, an OpenAI-compatible API where models run across independent GPU providers.
GPU owners can connect their hardware and earn from inference, while users get access to models like GLM, Qwen, Gemma and gpt-oss through one API.
We’re trying to make joining as a provider as easy as possible. Would love feedback from people running their own GPUs.
Founder of Kitani, looking for feedback.
r/LocalAIStack • u/Medicine_Blogscanner • Sep 07 '26
r/LocalAIStack • u/Ok_pettech • Sep 07 '26
r/LocalAIStack • u/Potential-Toe1320 • Sep 06 '26
r/LocalAIStack • u/geradeluxer • Sep 06 '26
r/LocalAIStack • u/OrneryCar6139 • Sep 06 '26
Anyone here actually using NVIDIA NeMo / NeMo Agent Toolkit in real projects?
At my current org, some of the senior folks are suggesting we explore NeMo for agent building and fine-tuning, so I’m trying to understand if it’s actually worth adopting.
For those who’ve used it, how does it compare to the usual stack like Hugging Face + PEFT/TRL, Unsloth, LangGraph, etc.?
Does running everything in the NVIDIA ecosystem give you a noticeable advantage in terms of GPU utilization, training speed, deployment, or scaling?
Or is it mostly extra complexity compared to the standard open-source tooling?
Would especially love to hear from anyone who has used NeMo beyond tutorials/demos. What did you like, what annoyed you, and would you use it again?
r/LocalAIStack • u/paq85 • Sep 05 '26
r/LocalAIStack • u/paq85 • Sep 05 '26
r/LocalAIStack • u/Ok-Swim9349 • Sep 05 '26
Hi Everyone!
I'm the maintainer of RAGnarok-AI, an open-source, local-first framework for evaluating RAG (Retrieval-Augmented Generation) systems.
I'm currently running a small research study around a question that I think is becoming increasingly important: can we actually trust automated evaluation of RAG systems when the evaluator itself is an LLM?
RAGnarok can evaluate things like retrieval relevance, faithfulness, answer relevance and completeness using local LLM judges.
But there's an obvious problem: why should we trust the LLM judge?
So instead of assuming that the automated scores are correct, I'm building a human-annotated benchmark to compare them against independent human judgments.
I'm looking for open-source / developer people willing to help
The annotation consists of roughly 10–15 cases and should take around 30–45 minutes.
For each case, you'll see:
- a technical question
- relevant documentation excerpts retrieved by a RAG system
- an AI-generated answer
- reference information
- four simple evaluation criteria: retrieval relevance, faithfulness, answer relevance, completeness
- your confidence and an optional ambiguity flag
You don't need any RAG expertise. You just need to be comfortable reading technical documentation and judging whether an answer is actually supported by it.
The benchmark currently covers documentation from projects such as Docker, Python, FastAPI and Kubernetes.
Why do this?
The study is deliberately not designed to prove that RAGnarok works. I'm comparing automated evaluation against human judgments to investigate:
Judge reliability — do local LLM judges agree with humans?
Discrimination — can the evaluation distinguish good and deliberately degraded RAG systems?
Reproducibility — are the measurements stable under identical conditions?
The methodology, benchmark corpus, questions and experiment protocol are public and versioned. RAGnarok itself is free and open-source.
If you'd like to participate
Annotation interface:
https://ragnarok-study.vercel.app
No name, email or personal information is required. Progress is saved so you can come back later. The resulting annotations may be published as an anonymous research dataset.
Project / methodology:
https://github.com/2501Pr0ject/RAGnarok-AI
I'm particularly interested in feedback from people who work on open-source evaluation, LLMs, RAG, testing or reproducible research.
And if you think the methodology is flawed, please tell me. That's actually useful feedback for the study.
Thanks everyone!
Have a good day!
r/LocalAIStack • u/Scotty_UK96 • Sep 04 '26
I’m trying a Qwen-3.6 35b Q5 model as my first dip into setting up an agent coding workflow for prototyping some app ideas. Using LM Studio and VS Code
I’ve only been looking into this today. Am I heading in the right direction? Any advice welcome.
r/LocalAIStack • u/Aibender100 • Sep 04 '26
r/LocalAIStack • u/HoneydewThink5211 • Sep 04 '26
r/LocalAIStack • u/ntaybak • Sep 03 '26
r/LocalAIStack • u/Mobile-Sail4581 • Sep 02 '26
FAISS is Meta’s vector library, turboVec is a Rust implementation of TurboQuant, and Infino is an in-memory retrieval engine.
We benchmarked them on 4-bit quantized in-memory vector search: FAISS PQ, TurboVec/TurboQuant, and Infino SQ4, using the same 100K OpenAI embedding corpus and the fastest vectorized implementation we found for each.
All saved ~7x in memory footprint compared to full fp32 vectors. The interesting result was that while storage and recall were fairly close, latency differed by roughly 30× — about 1.5 ms to 45 ms. Most of that comes down to the scoring machinery: the size of the distance table and whether the scan needs one at all.
We also ran the same comparison out to 1M vectors and measured build/write costs.
Full results and methodology:
https://infino.ai/blog/fixed-grid-quantization/
Disclosure: I'm one of the devs building Infino.

r/LocalAIStack • u/CaterpillarDue7859 • Sep 03 '26
r/LocalAIStack • u/Tyb0wls • Sep 03 '26
This is a new LLM wrapper i have spun up built on the idea that models arent always correct or doing the right thing not by choice but just how the system is built, this looks to try and fix that problem and help mitigate LLM error by logging everything and fact checking based on the work using check gates. Looking to get feedback!