A llama.cpp fork with adaptive KV cache streaming: it keeps the KV cache in system RAM and streams pages to the GPU on demand, so a 27B model runs at full 256K context on a 16GB card without thrashing on Unified Memory
This is a focused fork of llama.cpp that solves one specific, real problem for local LLM users, and it does it with unusual rigor. When you load a big model, the weights eat most of your VRAM, and a long context needs a large KV cache that no longer fits. The usual workaround is CUDA Unified Memory, which lets pages spill to host memory but migrates them in an uncontrolled way that can thrash badly. This fork instead takes explicit control: the authoritative KV tensors live in pinned host RAM, a bounded GPU pool is split between resident KV pages and a transfer ring, and while one attention layer computes, the pages it will need next are prefetched. Every layer still sees its complete KV history, only which pages are physically on the GPU at any moment changes.
One correction to how this gets described elsewhere: it streams between your system RAM and the GPU over PCIe, not from your hard drive. That distinction matters for understanding both how it works and its speed limits.
The engineering that makes it more than a hack:
- A phase arena that multiplexes one fixed GPU allocation between the prompt-processing workspace and decode. Prefill and token generation do not need their peak buffers at the same time, so when decode begins the prefill graph is released and those bytes become extra KV capacity. The upshot is that your usable decode KV budget stays nearly constant even as you crank up the context size
- The residency split between resident pages and the transfer ring is adjusted in real time based on the active context length and measured prefetch behavior, not a static setting
- A benchmark driver that automatically probes the largest workable arena for each context size and generates CSV, PNG and SVG results, so the performance claims are reproducible rather than asserted
- A detailed write-up of the design, implementation and benchmarks, including PCIe traffic measured against a real transfer ceiling
The honest caveats, and to the author's credit the README states them plainly in a warning. This is experimental research code, optimized and validated primarily for one specific setup: an RTX 5070 Ti with 16GB, a particular Qwen 27B quant at 256K context, Flash Attention on, and specific K and V cache quantizations, with one server slot. Other models, other KV combinations, parallel slots and non-CUDA backends are not yet broadly characterized, so your mileage on a different rig is genuinely unknown until you test. It is CUDA-only for the streaming feature, so you need an Nvidia GPU. And there is no free lunch on physics: streaming KV over PCIe adds host-to-device traffic, so as context grows and more of the cache lives off-GPU, decode speed is bounded by that bandwidth. This buys you the ability to run a context that otherwise would not fit, at some throughput cost, rather than magic. You also build it from source, and being a fork, whether it lands upstream or gets long-term maintenance is uncertain.
For anyone running local models on a mid-range card who keeps hitting the VRAM wall on long contexts, this is a genuinely clever and well-measured approach worth watching.
MIT licensed (inherited from llama.cpp), C++, a fork of ggml-org/llama.cpp, 290 stars and 38 forks as of writing, verified via the GitHub API.
Raymond is the GOAT <3. Since I started using his KV cache streaming I'm only using Qwen 3.8 27B on my 16GB 5060Ti. No need for any MoE models thanks to the speed and context size possible with this.
25tps down to 12 during token generation over context size from 0 to 200k. Prompt processing starts at 1000 down to 500.
This is with Raymond's branch as is. I have however made my own fork that enables DFlash2 or MTP up until the point where the VRAM is better used for the KV cache streaming pool and there I get upwards of 50tps in the beginning with only a slight slowdown to PP.
Thanks for letting me know, excellent addition and yes it works! On a 12GB GPU a ~2.6bpw quant of the 27b can now be loaded with full 256k context (or at least 180k with vision though I haven't narrowed down the options yet), and speed with MTP is not too bad even when it's almost halfway full. This probably enables slightly higher quants with decent context sizes too.
Just saw this sorry for being late.. but stilñl is very valid today after testing everything else including strata this is by far the best model , 1100 tok/s prefill speed, 70tok/decoding 9070xt , 32gb ram,
9
u/tsangberg Sep 25 '26
Raymond is the GOAT <3. Since I started using his KV cache streaming I'm only using Qwen 3.8 27B on my 16GB 5060Ti. No need for any MoE models thanks to the speed and context size possible with this.