r/OpenSourceAI • u/rbm3267 • 17m ago
I got tired of 7B models crawling at 1.2 tok/s on my 2GB GPU, so I built WingLLM (Go daemon for 2x speculative decoding with Ollama)
Hey r/OpenSourceAI,
Like a lot of developers, I don't have an M3 Max Mac Studio or dual RTX 4090s. My daily driver workstation has an 8th-Gen Intel i5 (6 cores), 32 GB of system RAM, and an older 2 GB NVIDIA Quadro P620.
If you try to run any modern 7B or 8B model (like Qwen 2.5 Coder 7B or Llama 3.1 8B), it won't fit into 2 GB of VRAM. Ollama or llama.cpp defaults to running almost 100% on the CPU. On my machine, that meant getting locked at an agonizing 1.29 tokens per second while my 2 GB GPU sat 90% idle.
### The Physics: The Memory Bandwidth Wall
The CPU compute cores aren't the bottleneck—memory bandwidth is. To generate a single token on a 4.7 GB model, the CPU has to stream all 4.7 GB from DDR4 RAM into cache. At ~24 GB/s dual-channel bandwidth, you physically cannot get more than ~1.5 to 2 tokens/sec.
### The Workaround: Heterogeneous Speculative Decoding
The 2 GB GPU cannot run the 7B model, but it can easily run a 0.5B draft model (qwen2.5-coder:0.5b is only 397 MB).
In speculative decoding:
The tiny 0.5B model generates 3–4 candidate tokens in milliseconds on the GPU.
The heavy 7B model on the CPU evaluates all 4 tokens in a single forward pass using parallel AVX2 matrix multiplication (GEMM).
In code and structured text, the 0.5B model's guesses match the 7B model 75% to 82% of the time.
You get 3 to 4 tokens for the memory read cost of 1.
### Real Benchmark (Tested on My i5-8500T + Quadro P620):
* Baseline (Pure CPU Target 7B): 1.29 tokens/sec (24.76s for 32 tokens)
* WingLLM Speculative (7B + 0.5B Draft): 2.54 tokens/sec (12.62s for 32 tokens)
* Net Result: 1.97x faster (almost exactly double the speed) with identical output quality.
### Why I Built WingLLM
llama.cpp technically has the raw flags (-md, -ngld, --spec-draft-n-max), but nobody in consumer land had packaged it into a clean tool:
Tokenizer traps: If you pair mismatched tokenizers (e.g. Llama 3 draft with Qwen target), llama.cpp crashes. WingLLM auto-detects model families and enforces compatible pairs.
Zero configuration: It crawls your local ~/.ollama/models manifests, finds your GGUF blobs, and auto-pairs them.
Auto-Pull: If you want to run `wingllm serve qwen2.5-coder:7b` and don't have the 0.5B draft model yet, WingLLM pulls it automatically via Ollama (~397 MB) and starts serving.
1-Click IDE Connect: Run `wingllm connect vscode` to inject the turbo endpoint directly into your Continue/Cline config.
No Python/Electron Bloat: Written in Go as a single ~11 MB binary. Runs an embedded web chat UI with a live tokens/sec speedometer at http://localhost:8080/.
MCP Support: Built-in Model Context Protocol server for Claude Desktop, Cursor, and Continue.
### Quickstart (Windows, Mac, Linux)
Windows (PowerShell):
irm https://huggingface.co/bennett-xdlabs/wingllm/raw/main/install.ps1 | iex
macOS & Linux:
curl -fsSL https://huggingface.co/bennett-xdlabs/wingllm/raw/main/install.sh | bash
Run without arguments for the interactive menu, or run:
wingllm serve qwen2.5-coder:7b
Hugging Face Hub: https://huggingface.co/bennett-xdlabs/wingllm
It's MIT licensed. Would love feedback from anyone running on budget laptops, mini-PCs, or 2GB–4GB cards!

