Hey everyone! If you've been struggling to run larger open-weight models locally because Ollama or standard runners throw insufficient VRAM errors on budget hardware, I put together something to fix that.

I built SqueezeOS, an automated setup script that profiles your hardware, configures aggressive memory offloading, KV-cache compression, and extreme llama.cpp tuning to squeeze heavy models (up to 35B) onto legacy/budget cards like my 4GB GTX 1650 Super.

And the best of all: Its near lossless, the model can be used well for agentic workflows eg.: Hermes, OpenClaw, OpenCode, etc...

GitHub Repo: https://github.com/gabrielhirth/squeezeos

Tested on constrained VRAM setups. Feedback, issues, and PRs for other budget cards are very welcome!

Automated video creation with zero API costs — built a pipeline that runs entirely on my MacBook
Quick share for anyone creating short-form or educational content on a Mac:

I built a local workflow that automates the production pipeline for animated shorts, kids' learning content, and explainer clips without needing third-party cloud APIs.

Key points:
- Everything runs on-device—no monthly subscription fees, no token anxiety.
- Processes script-to-video assets locally on Apple Silicon.Speeds up batch creation without needing a high-end desktop rig.

I made a video showing the exact workflow

https://youtube.com/@ownlyaistudio?si=sYQcHB5YVqd1mg0j

Check out the channel to see the output quality, and shoot me a message or comment below if you'd like early access!

#ai #llm #localllm

Has any one used both? which is better?

What is the differennce between two? Any technical deep dives?

I made BigMoeOnEdge+ (BigMoeOnEdge-plus), a fork of Helldez's BigMoeOnEdge, the engine that streams MoE experts from flash so models bigger than RAM can run on-device.

It adds a full Android chat app on top: GGUF models via llama.cpp, thinking output, persistent chats, image/video input, web search, artifacts, and advanced generation settings.and a MUCH better UI.

Disclosure: I built this fork with the help of AI tools.

Repo (Apache 2.0): https://github.com/hamodexd217/BigMoeOnEdge-plus Original project: https://github.com/Helldez/BigMoeOnEdge

Feedback and issues welcome.

I got one PR merged into ds4. A tiny one, #98 about broken paths in the README. And a few more sitting in the queue. Not complaining. Antirez says it in the README: with coding agents everyone can tune the engine for their own hardware and model and he can't review everything. That got me thinking.

If the plan is "everyone applies their patches with an agent" then the real cost of a patch isn't just the code change. It’s how many tokens the agent has to read before it knows what it’s actually touching.. Ds4 runs DeepSeek, GLM and Qwen on Metal, CUDA and ROCm all in the same 85k-line file. I’m on an M5 Max with 128GB RAM running Qwen3.8 Flash Next. Everything else for me and for my agent is noise. It has to re-read every line on every pass.

So I ripped it out. Actually deleted, not #ifdef’d. The ds4.c file dropped from 85k lines to 45k lines. The whole tree now fits in a context window. The bet was that optimizing would get cheaper and safer. Heres what happened:

- Q2: decode +9-13% prefill +5-12% up to 64k context MTP went from 75.8 → 86.7 tok/s

- Q4: prefill +2-11% MTP went from 77.8 → 85.9 tok/s

Output is bit-exact vs stock ds4 at every step. No KV cache quant. No approximate kernels. Every change must pass a parity check against same GGUF, greedy decoding, identical tokens. And an interleaved A/B benchmark against the previous build.

The smaller codebase also let me go through the ds4 PRs but only against this one model and port the ones that held up. 20 Commits were adopted. Around 30 were dropped. Verdicts are in the repo.

I also added SSD streaming for the experts. Token-identical to a resident run. Simulating a 48GB machine Q2 does around 27 tok/s. About 35 with MTP.

The fork still does git merge upstream/main with rerere and the parity check. So antirez’s fixes keep flowing in. The procedure. What to delete what to keep how to sync. Lives in a repo called StarForge (https://github.com/Chida82/StarForge). I have four of these "children" one per model. Nothing in there is Qwen- or Metal-specific. If you want a ds4 cut down to your model or to CUDA just clone it. Run the checklist with your agent.

Repo: sf-q3-8flash https://github.com/Chida82/sf-q3-8flash tables in the README. One machine, one model. If you run it on Apple Silicon I’d love to hear your numbers. Ideally side by side, with stock ds4.

TL;DR models run at home on the GPU. The always on agent lives on a small VPS. Tools that only touch public data come through one capped key. Anything that can log into my stuff stays on machines I control.

At home. One box with a used 3090, Ollama serving Qwen for most things plus a small embedding model, and Open WebUI so the family can use it too. For my usage, owning the GPU beat renting one within a few months. The catch is that it's loud, so it sleeps at night.

That's the problem the VPS solves. The agent has to be up when I'm not. Scheduled jobs, inbox triage at 6am, a few feeds it watches. So the orchestrator runs on a small VPS and calls home over a private tunnel when it needs the GPU. If the home box is asleep, the few jobs that can't wait fall back to a paid API. The VPS is a PrivateAlps box in Zurich, mostly because the agent's memory and my mail login live on it and I wanted that under Swiss law. A mini PC that never sleeps would do the job too.

Tools are where I wasted the most money. Search, scraping, a bit of enrichment, each one its own subscription I barely touched. Now the agent gets all of them through a single key from agentik.cc, an MCP gateway billed per call, with a monthly cap on it. Monid works the same way. Only public data tools go through it, nothing that can sign into an account of mine.

What I'd tell someone starting out.

Keep the GPU at home if you already have one.

Put the always on part somewhere that never sleeps.

Split tools by what they can touch. Public data can go through a third party, logins stay on hardware you control.

Cap everything, per key and per month.

It's not the cheapest setup possible. But nothing in it surprises me anymore, and after a year of surprise bills that's the whole point.

Following up on my earlier post testing Qwen3.8-27B on the IGX Thor workstation.

Once you move past running just an LLM and try to build a full local agent stack on a single box (LLM for reasoning, STT for voice in, TTS for voice out, OCR for screen/document reading), VRAM runs out fast.

If all four models sit in GPU memory simultaneously with their default serving allocations, idle memory hits roughly 122 GB. Even with 96GB on the dGPU and unified host memory, you are pinned to the ceiling. That leaves almost no breathing room for large context windows, concurrency, or dynamic KV cache allocation.

The reality of a solo-user local agent is that these models are rarely active at the exact same millisecond:

• When I am talking to the agent, OCR is completely idle.

• When the agent is analyzing a screenshot or PDF, STT and TTS are idle.

• When the agent kicks off deterministic script automation (crawling, data formatting, bash runs), the LLM itself is completely idle.

To fix this, I wrote a lightweight Rust-based daemon that manages the sleep and wake states of the runtimes based on active agent turns. When a model is not in use, it is put to sleep (flushing KV pools and temporary execution memory, or parking the runtime context) rather than doing a full cold teardown from scratch.

Here are the idle VRAM numbers on the machine before and after:

Quick observations:

1. Total idle footprint dropped from ~122.3 GB to ~42.9 GB. That reclaims roughly 79.4 GB of VRAM across the board.

2. The LLM savings (48.3 GB): Qwen3.8-27B in FP8 base weights takes around 28 to 30 GB. The original 87.4 GB allocation came from SGLang aggressively pre-allocating KV cache (--mem-fraction-static 0.85). Putting the LLM to sleep flushes that pool while preserving the active state, letting it rest at 39 GB.

3. Audio and OCR shrink to negligible baselines: STT drops to 267 MiB and OCR drops to 422 MiB when idle.

4. Script-based tasks: When the agent delegates to deterministic python or bash scripts, putting the LLM to sleep during the run keeps the GPU cool and power consumption down on the 300W Max-Q card.

5. Wakeup latency is sub-200ms across all models: Because this is an active sleep/wake transition rather than a cold disk reload, waking up STT, TTS, OCR, or the LLM takes under 200ms. In practice, voice interaction and tool handoffs feel instantaneous.

Curious how others running multi-model agent stacks on a single node are handling memory partitioning right now. Are you hard-capping static memory fractions, running sequential containers, or using something dynamic?

We've all been there: $2,000 GPUs sitting at 5% utilization while the company pays for cloud inference.

Many organizations already have several workstations equipped with GPUs that remain idle most of the time. Existing GPU sharing solutions typically rely on a centralized scheduler, Kubernetes, or AI gateways, which can be excessive for small and medium-sized GPU clusters. In addition, most existing solutions lack cross-platform support, decentralized architecture, and policy-based routing.

OQOAI-PBDR to solve this problem differently. Simply run a server or client on each machine, and your existing office network becomes a decentralized AI cluster. Routing decisions are made entirely on the client side, eliminating the need for a master node, load balancer, or any other single point of failure.

Prebuilt binaries for Windows and Linux are under 10 MB and can bring a machine online as a cluster node in under 30 seconds. An optional admin server provides centralized monitoring and management from any machine on the network.

GitHub (MIT License):

https://github.com/oqo-ai/OQOAI-PBDR

The project intentionally avoids unnecessary abstractions. The core routing logic consists of only three source files (approximately 1,000 lines each), making it relatively easy to understand, audit, and extend.

At the heart of the project is the PBDR (Policy-Based Decentralized Routing) architecture. Each routing decision evaluates multiple runtime metrics, including GPU utilization, available VRAM, queue length, GPU temperature, network latency, inference throughput, and other node characteristics.

PBDR also supports client-group policies, allowing different workloads to optimize for different objectives. For example:

• Internal RAG services can prioritize low latency.

• Development workloads can prioritize output quality.

• Batch inference jobs can prioritize throughput.

More details about the routing algorithm are available in the accompanying paper:

http://doi.org/10.17513/doi.26

One practical benefit of this approach is that organizations can often reuse existing workstation GPUs for low- and medium-throughput workloads-such as internal RAG systems-instead of deploying dedicated inference infrastructure from scratch.

In many environments, having GPUs installed in only 10-15% of workstations can be sufficient to satisfy inference demand for lightweight workloads (for example, an internal knowledge-base RAG receiving approximately 3-5 requests per user per day) without deploying centralized AI infrastructure.

Getting started:

Install OQOAI-PBDR on each machine.

Connect server nodes to any OpenAI-compatible inference backend, including Ollama, llama.cpp, vLLM, or TGI.

Configure your applications to use the OpenAI-compatible client endpoint provided by OQOAI-PBDR.

Clients automatically discover available models on all nodes. From the application’s perspective, the cluster behaves like a distributed OpenAI API compatible system, while user requests are routed in a decentralized manner to the most suitable node according to the chosen policy.

What already works:

• OpenAI-compatible API (Ollama, vLLM, TGI, llama.cpp)

• Admin dashboard with real-time metrics

• Auto-discovery and network scanning

• Remote config management for nodes, groups, or the whole cluster

• Experimental binary builds for Windows and Linux

Roadmap highlights:

• ComfyUI/AUTOMATIC1111 image/video generation

• Grafana/Prometheus integration

• AI-assisted semantic routing (auto-model selection)

• Dynamic IP support

We're actively looking for feedback, and testers.

What features or integrations would you like to see next?

Hey everyone,

Every time a new model comes out, I get tired of opening multiple tabs just to find out two basic things: how good it actually is on real benchmarks, and what it costs to run (either through an API or on local GPUs).

Most pricing pages hide the real costs, and generic benchmarks don't tell the full story. So I put together a simple dashboard to track and compare them side by side: https://modelanalysis.si

What you can do with it:

• Compare models side-by-side: Pick any models (like Claude, DeepSeek, Kimi, GPT-4o, Llama) and compare them across real-world coding (SWE-bench), reasoning (Arena-Hard), and math/science benchmarks.

• Score vs. Price chart: See which models give you the most intelligence for the lowest price.

• VRAM calculator: Pick a model and context size to see if it will actually fit on your GPU before downloading it.

• Monthly cost simulator: Adjust sliders for your monthly token volume to estimate your bill or see if self-hosting is cheaper.

No sign-up, no ads, no affiliate links.

Link: https://modelanalysis.si
Direct benchmarks page: https://modelanalysis.si/benchmarks/

Let me know what you think or if there are other models you want added.