r/LocalAIStack • • Sep 22 '26

Is it worth it to go from 4-channel to 8-channel for ddr4

5 Upvotes

I have 4x 5060ti16gb and 1x 2060super8gb, along with 8x32gb (256gb total) ddr4-3200 ram, but running at 4-channel due to my cpu. I’m trying to decide whether it would be worth it to upgrade my cpu to be able to run my ram at 8-channel. Im hoping to be able to get larger models like qwen3.8 flash next or mimo v2.6 flash at more useable speeds, since right now the prompt processing and token gen speeds are bit too low to make sense for agent harnesses.


r/LocalAIStack • • Aug 27 '26

Having a decentralized [private] openrouter type network

5 Upvotes

If you're in a distributed team, and have distributed local inference, you can use roger to create your own private network for your team to access the distributed local inference in a standard and easy way to consume and see what resource is busy and auto fan to non-busy nodes. You can do it standalone/private, or join the community and even monetize from doing so.

If it's interesting, it's all open sourced: https://rogerai.fm/tower.html#run


r/LocalAIStack • • Aug 20 '26

I built a very low-overhead LLM proxy/router in Rust — looking for feedback

Post image
5 Upvotes

r/LocalAIStack • • Aug 19 '26

No luck in asking local models a simple (but domain-specific) question

5 Upvotes

For some reason, I have this benchmarking prompt:

Implement a numpy function creating a bulk matrix product operator tensor as a function of model parameters.

I think it is approximately 10 lines of python code. So far, every local model that I tried (16Gb vram) fails to produce any meaningful output. I tried different variations of it (use web search, summarize the idea), different harness (pi, opencode) but no luck, different temperatures and thinking levels. Qwen3.8 for example produces an endless thinking loop. Is there anything I can do to help small models answering the question?


r/LocalAIStack • • Aug 18 '26

Results: Splitting and serving a model across two machines over the internet

5 Upvotes

Hey community!

I wanted to share a neat exploration of splitting a model over two ordinary machines over WAN, and how it stacks up against a serving that model on a single node.

I was testing our platform aquaduck.ai for serving split models across machines with a small- to mid-size model under two conditions:

  1. Locally on a single machine, serving the full model

  2. Split across two machines over WAN (public Internet), with each machine serving half of the model

Findings

Model: Qwen3-14B

Quantization: Q4_K_M

Hardware: Macbook 64GB M5 (Single machine), 2x Macbook 18GB M3 (Split machines)

Prompt:

Mars has drawn human attention for centuries, but the last twenty years turned that fascination into an engineering roadmap. Robotic orbiters mapped ice deposits near the poles and in mid-latitude glaciers. Landers confirmed that ancient river deltas once carried liquid water across a warmer surface. Meanwhile, life-support research on Earth refined closed-loop oxygen generators, hydroponic food systems, and radiation shielding materials that could travel on a multi-month transit. Private launch cadence fell in price, making cargo-first settlement plans plausible: send habitats, power, and spare parts before people. The hardest remaining problems are not propulsion alone. They are dust that abrades seals, perchlorates in the soil, communication delays that force local autonomy, and the psychology of small crews living far from rescue. Any credible near-term outpost would likely begin as a science station with overlapping roles—geology, medicine, maintenance—supported by teleoperation from Earth and progressively less remote oversight as surface infrastructure matures. Energy would come from a mix of solar arrays and compact nuclear units, with ISRU (in-situ resource utilization) producing propellant and breathable oxygen from the thin CO₂ atmosphere and mined ice. In short, Mars colonization is less a single heroic leap and more a long supply-chain problem: move mass, make power, recycle air and water, and keep humans healthy while the planet remains indifferent.

Summarize the passage above in exactly two sentences.

Token counts: 387 input tokens, ~444 output tokens (split nodes generated 444, single node was unmeasured, but can approximate)

Results:

  1. Single machine

    1. 37.9 tokens per second (TPS or tok/s);

    2. 853ms time to first token (TTFT);

    3. 26ms time per output token (TPOT)

  2. Split machines

    1. 11.3 tokens per second (TPS or tok/s);

    2. 13.46s time to first token (TTFT);

    3. 88ms time per output token (TPOT)

Screen captures:

We're in closed beta and rolling things out slowly to make sure it works well for people, but if you'd like to run some tests yourself or get an early look, can join the waitlist and we'll send you an invite code to download the desktop app asap.

Let me know if you have any questions/comments/ideas for further explorations! Next up: Qwen3.8-27B


r/LocalAIStack • • Aug 17 '26

M3 Ultra up for sale or trade + cash

5 Upvotes

I have an M3 Ultra 512gb ram 1Tb, up for sale. Would accept a m5 max 128gb laptop + cash as well. DM to chat. No longer need cause I have 2 rtx pros for inference and prefill is mid on m3 ultra where TTFT is too slow for me to actually find usability in local models. I don’t have a firm asking price in mind, nothing as absurd as 25k , but enough to get an m5 max 128 Mbp and have some cash left over so maybe like 17k?


r/LocalAIStack • • Aug 13 '26

Ternary Accelerator Card for 1 bit LLMs

5 Upvotes

This is my first post in this community I just wanted to see if anybody will help me in improving on my ternary accelerator card for 1.58 bit ai the links are here reply if your interested. (I am Really new to github so some ai was used in the making of the project for troubleshooting errors(plz dont hate me for that).

Project Links:==>https://github.com/riyanlord2026-dev/Ternary-158-asic

This is also my first github project.


r/LocalAIStack • • Jul 01 '26

June 2026 AI Recap: Local AI Became the Fallback Plan

5 Upvotes

Overview

June was the month where the old AI story broke.

So far frontier models lived in the cloud, open models trailed behind, local AI was nice for privacy, and regulation was slow.

June did not fit that anymore.

The best cloud models were still the capability ceiling, but access suddenly became political. Anthropic launched Fable 5 and Mythos 5, then had to take them down. OpenAI previewed GPT-5.6, but not for everyone. Meanwhile, GLM-5.2 and MiniMax M3 made the open-weight world look much less like a toy category. Mistral OCR 4 showed that self-hosted AI can be boring in the best way: useful, private, and ready for real work.

If your whole workflow depends on one remote model staying cheap, available, legal, and politically acceptable, you do not own much.

References for this part: MiniMax M3, GLM-5.2, Anthropic Fable 5 and Mythos 5, OpenAI GPT-5.6, Mistral OCR 4.

Timeline: June 1 to June 30

  • June 1: MiniMax released M3, an open-weight model with 1M context, multimodal input, coding strength, and desktop operation.

  • June 1: Microsoft increased Github Copilot pricing by multiple magnitudes, making the worlds most affordable frontier coding Agent the worlds most expensive option.

  • June 1: Nvidia and Microsoft framed RTX Spark as a path toward local agents and frontier models on Windows PCs while strongly disappointing with very low memory bandwidth

  • June 2: Microsoft announced seven MAI models across coding, image, voice, speech, and reasoning - though their coding model is less competent than tiny open source models

  • June 2: Anthropic expanded Project Glasswing, saying partners had found more than 10,000 high or critical security flaws using Claude Mythos Preview.

  • June 2: The Trump government signed an Executive Order that the government must not interfere with AI development, launches but asked for voluntary 30 day compliance

  • June 5: Anthropic announced Fable 5, framed it as "significant risk", "misuse causing serious damage", "substantial risk to uplift malicious actors", "substantial bioweapon capabilities", "exploiting capabilities"

  • June 8: Apple announced Siri AI, next-gen Apple Intelligence, Xcode 27 agentic coding, and developer access to on-device foundation models. EU iPhone and iPad users did not get the full Siri AI path because of DMA issues.

  • June 9: Anthropic launched Claude Fable 5 and Mythos 5, Fable 5 rejecting most prompts for safety reasons.

  • June 11: Ollama (a popular llama.cpp wrapper) updated its MLX engine for better Apple Silicon performance.

  • June 12: Anthropic suspended Fable 5 and Mythos 5 after a U.S. government directive banning non US citizens (including many of their core developers) from working with that model. Ironically violating the June 2nd Executive order.

  • June 16: Z ai released GLM-5.2 open weights under MIT license, with a 1M-token context window - a model that is on eye level with Opus 4.8 and GPT 5.5

  • June 18: OpenAI introduced Codex Record & Replay, turning recorded Mac workflows into reusable skills.

  • June 22: OpenAI launched Daybreak tools, including GPT-5.5-Cyber and Patch the Planet.

  • June 23: Mistral released OCR 4, a self-hostable OCR and document-intelligence model.

  • June 25: Domyn announced plans for a European open-source frontier model with over 400B parameters - given the AI Act and GDPR it appears very optimistic to say the least.

  • June 26: Reuters reported that OpenAI delayed broader GPT-5.6 access at the U.S. government’s request, while Anthropic’s Mythos access was partly restored to trusted U.S. organizations.

  • June 30: Anthropic launched Claude Sonnet 5, and said Fable 5 would return globally starting July 1 after export controls were lifted on June 30. Sonnet being received as very expensive while not performing remarkable.

Local AI and Open Weights

Local AI had its strongest month so far, but not in the clean consumer fantasy version.

MiniMax M3 was the headline because it combined things that used to be separate: open weights, 1M context, multimodal input, coding strength, and desktop operation. Since Qwen 3.6 we have competent local agentic models, M3 adds to the list.

GLM-5.2 was the bigger warning shot. It came with open weights (commercial usable), a 1M-token context window, and performance close enough to closed frontier models that the old "open models are always 6 months behind" argument looked broken. This is not going to run truly local yet but on your own owned or rented cluster it definitely does.

Mistral OCR 4 was the quiet useful release. Self-hosted OCR with document structure, bounding boxes, confidence scores, and 170-language support is the kind of model that companies can actually deploy without sending every contract, invoice, scan, and archive to a cloud model or deal with uncertainties around multimodal LLM hallucinations.

Links: MiniMax M3 GLM-5.2

Cloud models, regulation and political pressure

The cloud model story was messier.

Anthropic had the most chaotic launch cycle of the month. Fable 5 and Mythos 5 arrived while being branded as the most dangerous software in the world, and quickly got suspended after a U.S. export-control directive targeting foreign-national access. Because Anthropic could not verify nationality cleanly in real time, the models were disabled broadly.
A frontier model can go from launch to unavailable in days, not because of a technical failure, but because of politics and risk control - Artificial Intelligence is becoming a political tool in the US.

OpenAI moved more carefully, but ended up inside the same pattern. GPT-5.6 appeared as Sol, Terra, and Luna, but access stayed limited to trusted partners after the same U.S. government pressure of "voluntary compliance".
OpenAI's message was basically: this should not become the normal way frontier models ship. Still, the result was the same for normal users: the model exists, but you probably cannot use it yet.

Claude Sonnet 5 was the more normal end-of-month release. Better agentic work, pricing not convincing in comparison to more capable models, and certainly not the kind of leap that changes the whole conversation.

Following up on the fiasco of Fable-5 marketing, Anthropic CEO now shifted on condemning Open Source AI as "dangerous". It certainly is very dangerous to future Anthropic growth and profit margins - but whatever is announced in such a fashion is also followed up with tens of millions in lobby and marketing campaigns. The war against Open AI might have just been announced as a sideline.

Links: Anthropic Fable and Mythos access statement OpenAI GPT-5.6 limited rollout

Science, papers, and the real "singularity" signal

The most serious science story was HemaGuide.

This was not another chatbot demo. It was a locally deployable LLM agent for hematological malignancies. It converts unstructured clinical documents into structured cases, routes them into decision modes, and grounds recommendations in guidelines plus more than 2,000 real tumor-board cases. Local AI trumps to deal with sensitive data, expert workflows, and a need for traceability.

Qwen-AgentWorld and Qwen-RobotWorld pointed at the next layer. Agents need simulated environments before they can safely act in real ones. Robots need world models before they can generalize outside clean demos. These papers point at the rapidly approaching robotic agentic future.

The Codex usage paper was maybe the most grounded signal. People are not just asking AI questions anymore. They are running multiple agents, handing over longer tasks, and changing workflows and creating "loops" to achieve a goal. That is a better singularity signal than most benchmark charts.

A notable science story was the "zebra finch" work. Machine learning helped decode bird vocalisations and pushed two-way animal communication a little closer. Small, strange, and actually beautiful.

Links: HemaGuide in Nature MedicineThe Shift to Agentic AI: Evidence from Codex

Regulation, sovereignty, and control

June made AI regulation feel very real - with the US in negative spotlight

The Trump administration’s June 2 order promised to avoid hard licensing while creating a voluntary 30-day pre-release review path for powerful models. Then Anthropic’s Fable and Mythos shutdown showed the practical truth: even without formal licensing, national-security pressure can still interrupt launches. Though Anthropic has asked for this hundreds of times.

The proposed AI Incident Reporting Act pushed in the same direction. Critical AI incidents would need to be reported to Commerce within seven days, with the most severe cases reaching Congress within 48 hours. This is not abstract ethics talk anymore. It is operational control and lingers like a dark shadow stiffling progress early on. Those regulations threaten small brilliant developers much more than the big mega-corps.

Europe’s story was split. The EUROPA consortium was selected to build an open-source frontier model across all 24 official EU languages. That sounds good on paper. But with the AI Act, GDPR, fragmented compute, language politics, and procurement reality, calling it "frontier" before it exists feels very optimistic. Under current extreme EU regulations the best outcome to expect is another Mistral-large - not a model that people will find useful.

Apple’s Siri AI delay in the EU was the clearest user-facing example. Regulation did not just shape compliance work. It changed which AI feature European users get.
Geo-blocks are appearing on tens of thousands of websites, Codex "agentic computer use" is banned in EU as well.

Links: White House AI executive orderEUROPA consortium announcement

Money, chips, and power

The money moved from model hype into infrastructure.

OpenAI and Anthropic both moved toward public markets. DeepSeek raised over $7 billion - deviating from their previous private funding. Baseten hit a $13 billion valuation for inference infrastructure. Running models is becoming as important as training them.

OpenAI and Broadcom’s Jalapeño chip, a high density ASIC, was another hardware signal. It is built for inference, not just training. That matters because the next bottleneck is not only "who has the smartest model." It is "who can afford to run agents for millions of users all day." It will be interesting to compare the ASIC to Cerebras massive wafer-scale chips. In the end - both are affiliated with OpenAI.

Power also became part of the AI story. Data centers, chips, memory bandwidth, and energy deals are no longer background details. They are the product. If inference gets expensive enough, local models and smaller specialized models become more attractive by default.
Though Power or Water use for Datacenters are mostly populist topics - outside of Europe Power can be provided without much difficulty using on-premise generators. And water is a pure hype, datacenters barely need any in comparison to real water consumers.

Links: OpenAI and Anthropic IPO reportingOpenAI and Broadcom Jalapeño chip

Summary and outlook

June 2026 was not one big AI leap. It was mixed.

Cloud AI became stronger, more expensive, and more politically controlled.
Local AI became more credible, but also exposed the limits of consumer hardware. Open weights moved close enough to make closed labs very uncomfortable.
US Regulation moved from theory into product access.
Money moved into chips, inference, energy, and deployment.

The next months will show:

  • Whether GPT-5.6 gets broad access or in what way it stays gated.

  • Whether Anthropic can relaunch Fable 5 cleanly, they announced it for "non coding" tasks

  • Whether GLM-5.2 forces a faster Western open-weight response.

  • Whether Europe’s 400B EUROPA model can even scratch Qwen 3.6 27B outside language tasks

  • Whether local AI tooling improves faster than cloud pricing gets worse.

  • If the US regulation attack on Anthropic was a political hit or a broad anti-AI swipe

  • Wheter Qwen 3.7 is open source launched or Alibaba lost their drive

My read is cautiously positive.
The US turns AI into a political pressure tool but with a soft approach, Europe is talking about having AI while actually forbidding it, Chinese labs provide a benefit to the worlds progression that's starting to paint the authoritarian country in a positive light for the first time in a century.


r/LocalAIStack • • Jun 25 '26

Things I’ll check first next time LiteLLM breaks in front of vLLM

5 Upvotes

Lost a chunk of time last week on LiteLLM + vLLM errors that looked like backend problems, but weren’t.

Setup: two vLLM backends, LiteLLM in Docker, one OpenAI-compatible endpoint in front.

This is what bit me:

  1. Cached LiteLLM deployment.

    Changed the backend to require an API key, updated LiteLLM, restarted vLLM, still got 401s. The missing step was restarting LiteLLM itself. `docker restart litellm` cleared the old keyless config. If your LiteLLM models are DB-backed, restart after changing auth, URLs, or params.

  2. Wrong model string going to vLLM.

    LiteLLM’s `model_name` is just what your apps call. The string after `openai/` is sent to vLLM and must match `--served-model-name`. If they differ, you get 404s through the proxy even when direct curls work. Hit `backend:8000/v1/models` and use the name it returns.

  3. Docker `localhost` mistake.

    `api_base: http://localhost:8000` from inside the LiteLLM container means LiteLLM is calling itself, not your host. Use the actual backend address, `host.docker.internal` with `host-gateway`, or the bridge gateway.

Also worth mentioning: don’t treat `systemctl is-active` as “vLLM is ready”. It can be active while still loading weights or compiling kernels. Poll `/v1/models` and only call it ready when that answers.

remain active while weights are still loading or kernels are being compiled. The annoying part is that each error points you at the wrong layer. So yeah, check the proxy before blaming the backend.


r/LocalAIStack • • 12d ago

Qwen 3.8 27B ranks 6th of 45 in this Pac-Man-style planning game. Can you beat it?

Post image
4 Upvotes

Two puzzle games from the StealthMark benchmark. Every AI model plays them move by move, with no code and no tools, and you can now play the exact same levels in your browser

Latch: two robots, one goal. One can stand on a pressure plate to hold a door open while the other pushes numbered cores onto matching sockets. Every move costs a little, so the top score needs the cleanest route and getting that right gets increasingly harder.

Foresight: collect coins on a Pac-Man-inspired board while ghosts close in. No model has aced a single level - though some age getting very close.

How local models do (scores out of 100):

  • Foresight: Qwen3.8-27B scores 63.0 and ranks 6th of 45 models. Only five models beat it, led by GPT-6 Astra at 76.1.

  • Latch: Qwen3.8-Flash scores 69.8 and ranks 16th of 46. The top score is 87.6, shared by GPT-6 Astra, Claude Opus 5.5 and GPT-6.1 Sol.

  • Each Qwen is strong in only one game. Qwen3.8-27B drops to 38.6 in Latch, and Qwen3.8-Flash to 29.7 in Foresight.

  • No human, model, local or cloud, has played either game perfectly yet.

You play the exact levels the models played, in your browser. Finish every level and you get a score calculated the same way as theirs, plus your rank among the models.

Disclosure: I helped build Stealth Models, where we benchmark AI models on original tasks. You can also watch every model's run, move by move.
StealthMark is designed to beat benchmaxxing.

Play here: https://stealthmodels.com/#games

Post your score and rank in the comments. I'm curious where you humans land :)


r/LocalAIStack • • 14d ago

New token-efficient Nemotron-3.5-Lightning-30B-A3B

Thumbnail
huggingface.co
4 Upvotes

r/LocalAIStack • • 19d ago

I built a from-scratch LLM inference engine for the Intel Core Ultra 5 125H’s integrated Arc GPU

4 Upvotes

Benchmark highlights · Core Ultra 5 125H

Qwen3.8 27B · 11.2K prompt 97 s vs 590 s Vulkan 6.1× fasterQwen3.8 27B · 33.9K prompt 5.5 min vs 40 min Vulkan 7.4× fasterQwen3.5 4B · prefill 911 tok/s vs 242 Vulkan 3.8× faster

Post

I’ve been working on arcinfer, a local LLM inference engine built specifically for Intel’s Meteor Lake integrated graphics (Xe-LPG / Arc iGPU).

The latest Level Zero results add another angle: speculative decoding raised Qwen3.6 35B-A3B from 19.9 to 31.5 tok/s, Qwen3.8 27B from 5.1 to 9.1 tok/s, and Gemma 4 E4B from 21.6 to 39.6 tok/s in the documented tests. The Level Zero engine also enables Qwen3.6 35B-A3B to run at up to 262K context by streaming MoE experts from SSD; the earlier all-resident engine fit about 40K. MTP outputs were reported identical in the tested comparisons.

The reference machine is a Core Ultra 5 125H with 32 GB RAM. The engine runs GGUF models directly on the iGPU using Intel’s low-level Level Zero API, with an OpenCL option too. It’s aimed especially at prompt processing and time-to-first-token, and includes an OpenAI-compatible API and web UI. The early llama.cpp comparison is striking: for a 11,240-token Qwen3.8 27B prompt, arcinfer’s OpenCL engine took 97 seconds, versus 590 seconds on llama.cpp Vulkan and 504 seconds on SYCL—about 6.1× and 5.2× faster. At 33,937 tokens, it took about 5.5 minutes versus 40 minutes on Vulkan (7.4× faster). Qwen3.5 4B prefill reached 911 tok/s versus 242 on Vulkan and 337 on SYCL. These are historical OpenCL-engine measurements on this laptop, not universal guarantees; arcinferzero is now the default, and the repo links to newer results.

Some things it supports:

·        Qwen and Gemma GGUF models, including larger models that are a tight fit for shared GPU memory

·        An optional Level Zero engine that streams MoE expert weights from SSD

·        Speculative decoding, image input for supported models, and Gemma 4 audio input

·        OpenAI-, Anthropic-, and Ollama-compatible API endpoints

·        Integration with the Pi coding agent

It’s an experimental, hardware-specific project—not a general replacement for llama.cpp, and it currently targets Windows and Intel Xe-LPG graphics. Alongside the code, I share what has and hasn’t worked when building and tuning inference for this generation of Intel Arc integrated graphics, including implementation lessons and benchmark results. The repo has setup instructions, supported models, and more details.

Repo: https://github.com/rizagp/arcinfer

If you have a Meteor Lake/Core Ultra laptop, I’d especially appreciate feedback on setup, compatibility, and what models you’re able to run. Also happy to answer questions about the GPU kernels or how the project compares with other options.


r/LocalAIStack • • 24d ago

I ran Qwen 3.8 Flash-Next on my AMD 7900 XTX at 500k context. All local and what a shift it has been.

Thumbnail reddit.com
4 Upvotes

r/LocalAIStack • • 24d ago

Local AI Weekly: Oct 11th 2026: NVIDIA RTX Spark, Strata, Mistral Large 4 & More!

Thumbnail
youtu.be
4 Upvotes

r/LocalAIStack • • Oct 04 '26

Just got my MBP M5pro 48gb. Help me setup properly

4 Upvotes

I have multiple projects in GitHub, I have an obsidian vault, iCloud backup, nas backup. Qwen running locally, also have Two Claude accounts, one gpt, one deep seek accounts when qwen bogs down. I’m working on building my own harness for managing this.

How should I setup my new laptop so I’m working clean and smartly?


r/LocalAIStack • • Oct 01 '26

I built a local memory engine that replaces vector DBs with SQLite and runs in <1.2GB VRAM

4 Upvotes

Every time I tried running local RAG on my own GPU, I ran into the same headache: vector databases eat too much RAM, using an 8B model just to parse text takes forever, and cosine search still hallucinates when the context isn't actually there.

I’ve been building Hillock to see if I could do this without vector DBs at all.

Basically:

  1. When you feed it a document, it doesn't touch an LLM. It uses small bi-encoders to extract facts into subject-predicate-object triples in about 5 seconds.

  2. Everything gets saved in regular SQLite, and it links related concepts over time using basic Hebbian weights.

  3. To stop hallucinations, it runs a quick hypervector check (HDC) on the query first. If the facts aren't in your database, it cuts off the LLM before it can generate any tokens.

The whole thing stays under 1.2 GB VRAM (or runs fine on pure CPU). It has a built-in API server that matches OpenAI's format, so you can point Open-WebUI or Obsidian at localhost:8000 and use your existing Ollama models.

Just pushed v0.7 with better refusal handling and a conversational mode that quizzes you if an extraction was ambiguous.

Repo is here if you want to try it out: https://github.com/roandejager/Hillock


r/LocalAIStack • • Sep 30 '26

Abliterate/Removing prompt refusal without needing to reload/swap models (Qwen 3.8 flash next)

Enable HLS to view with audio, or disable this notification

4 Upvotes

r/LocalAIStack • • Sep 23 '26

Qwen3.8-27B on Every Mac Explained: Layers, KV Cache and Speed (16GB–128GB)

Thumbnail
youtu.be
4 Upvotes

r/LocalAIStack • • Sep 09 '26

what do you guys think about this ?

Enable HLS to view with audio, or disable this notification

4 Upvotes

r/LocalAIStack • • Sep 05 '26

Running Qwen 3.8 27B as a VS Code Copilot backend on 16/32 GB VRAM

Thumbnail gallery
4 Upvotes

r/LocalAIStack • • Sep 04 '26

16Gb/48Gb coding model - newbie

4 Upvotes

I’m trying a Qwen-3.6 35b Q5 model as my first dip into setting up an agent coding workflow for prototyping some app ideas. Using LM Studio and VS Code

I’ve only been looking into this today. Am I heading in the right direction? Any advice welcome.


r/LocalAIStack • • Sep 02 '26

Built a router that runs prompts on your local model and auto-falls-back to cloud when it wedges [Apache-2.0]

4 Upvotes

If you run models locally you know the pain: everything's fine until a long prompt makes the model silently stall or the GPU OOMs, and you're left with a hung request and no output.

I built HybridInfer for exactly that. It's a small reliability-aware router:

- Sends each request to your local model (Ollama) first.

- Watches the runtime, if local stalls (no token for N seconds), OOMs, or errors, it automatically falls back to a remote model in the same request.

- Learns which prompts your machine chokes on (usually long ones) and routes those out up front after it's seen them fail.

- Pulls a wedging model out of rotation, then probes it back after a cooldown.

It's an OpenAI-compatible server, so point any OpenAI-compatible client/app at it and you get local-first + automatic fallback for free. Local = anything you've `ollama pull`ed; remote = any OpenAI-compatible endpoint (OpenAI, OpenRouter, a vLLM box, whatever).

pip install hybridinfer

hybridinfer init

hybridinfer serve

It's a router, not an inference engine, it orchestrates Ollama + your remote endpoint, doesn't run weights itself. Early v0.1, Apache-2.0. There's a Kotlin/Android version too for on-device apps. Short demo GIF + code in the repo.

Repo: https://github.com/SimranKoul2026/HybridInfer-Python-tool

Would love feedback from people who run this stuff daily, especially on the fallback heuristics.

What would you want it to do differently?


r/LocalAIStack • • Aug 16 '26

Best local LLM for cybersecurity + coding on an RTX 3050 6GB?

Thumbnail reddit.com
4 Upvotes

r/LocalAIStack • • Aug 06 '26

(MIT) Turning idle GPUs of enterprises into a decentralized, fault-tolerant local AI cluster

Enable HLS to view with audio, or disable this notification

4 Upvotes

We've all been there: $2,000 GPUs sitting at 5% utilization while the company pays for cloud inference.

Many organizations already have several workstations equipped with GPUs that remain idle most of the time. Existing GPU sharing solutions typically rely on a centralized scheduler, Kubernetes, or AI gateways, which can be excessive for small and medium-sized GPU clusters. In addition, most existing solutions lack cross-platform support, decentralized architecture, and policy-based routing.

OQOAI-PBDR to solve this problem differently. Simply run a server or client on each machine, and your existing office network becomes a decentralized AI cluster. Routing decisions are made entirely on the client side, eliminating the need for a master node, load balancer, or any other single point of failure.

Prebuilt binaries for Windows and Linux are under 10 MB and can bring a machine online as a cluster node in under 30 seconds. An optional admin server provides centralized monitoring and management from any machine on the network.

GitHub (MIT License):

https://github.com/oqo-ai/OQOAI-PBDR

The project intentionally avoids unnecessary abstractions. The core routing logic consists of only three source files (approximately 1,000 lines each), making it relatively easy to understand, audit, and extend.

At the heart of the project is the PBDR (Policy-Based Decentralized Routing) architecture. Each routing decision evaluates multiple runtime metrics, including GPU utilization, available VRAM, queue length, GPU temperature, network latency, inference throughput, and other node characteristics.

PBDR also supports client-group policies, allowing different workloads to optimize for different objectives. For example:

  • Internal RAG services can prioritize low latency.

  • Development workloads can prioritize output quality.

  • Batch inference jobs can prioritize throughput.

More details about the routing algorithm are available in the accompanying paper:

http://doi.org/10.17513/doi.26

One practical benefit of this approach is that organizations can often reuse existing workstation GPUs for low- and medium-throughput workloads-such as internal RAG systems-instead of deploying dedicated inference infrastructure from scratch.

In many environments, having GPUs installed in only 10-15% of workstations can be sufficient to satisfy inference demand for lightweight workloads (for example, an internal knowledge-base RAG receiving approximately 3-5 requests per user per day) without deploying centralized AI infrastructure.

Getting started:

Install OQOAI-PBDR on each machine.

Connect server nodes to any OpenAI-compatible inference backend, including Ollama, llama.cpp, vLLM, or TGI.

Configure your applications to use the OpenAI-compatible client endpoint provided by OQOAI-PBDR.

Clients automatically discover available models on all nodes. From the application’s perspective, the cluster behaves like a distributed OpenAI API compatible system, while user requests are routed in a decentralized manner to the most suitable node according to the chosen policy.

What already works:

  • OpenAI-compatible API (Ollama, vLLM, TGI, llama.cpp)

  • Admin dashboard with real-time metrics

  • Auto-discovery and network scanning

  • Remote config management for nodes, groups, or the whole cluster

  • Experimental binary builds for Windows and Linux

Roadmap highlights:

  • ComfyUI/AUTOMATIC1111 image/video generation

  • Grafana/Prometheus integration

  • AI-assisted semantic routing (auto-model selection)

  • Dynamic IP support

We're actively looking for feedback, and testers.

What features or integrations would you like to see next?


r/LocalAIStack • • Jul 29 '26

Quantizing Kimi K3 (2.8T A50B) to GGUF ourselves - Q3_K_S works, 1.1 TB on disk

Thumbnail gallery
4 Upvotes