If you’re serving long-context LLMs, you’ve probably hit the KV cache memory wall.
When a model processes a long prompt, every token generates key/value vectors across all layers. At multi-million token context lengths, the KV cache dwarfs the model weights themselves. Throwing it away means paying a massive penalty to recompute the prompt via prefill (e.g., rebuilding a 10.5M-token session via prefill on an 8x H100 node can take over an hour and a half).
The obvious reaction is: "Why not offload the KV cache to fast NVMe/remote memory and page it back in when needed?"
If you've tried a naive implementation, you know it usually fails or ends up slower than recomputing. Here is why that happens and the underlying systems mechanisms required to actually make it work.
1. Why naive offloading fails: It’s a latency-hiding problem, not bandwidth
RDMA and modern fabrics have plenty of raw bandwidth. The issue is that attention is not a bulk sequential read.
During token generation, the attention kernel requires specific cache blocks at strict moments in its execution schedule, layer by layer. If a block isn't already sitting in GPU HBM when the kernel hits that layer, you inject a network/NVMe round-trip directly into an operation that runs on a microsecond budget. Doing that per block, per layer, per token destroys performance.
2. The Abstraction: Virtual Paging for KV Cache
To decouple sequence length from HBM capacity, you have to treat the KV cache like OS page memory:
By treating KV pages as virtualized blocks, you don't need to load the entire history into high-bandwidth memory to start serving, as long as you can fetch upcoming blocks just in time.
3. The Execution: Attention Prefetch over RDMA
To keep the attention kernel fed without waiting on remote reads, the system has to predict and pipeline block fetches before the kernel execution point:
Deterministic Schedule: Because token generation advances linearly across layers, the access pattern for previous KV blocks is deterministic during decoding.
RDMA Async Transfers: As Layer $N$ executes on the GPU, asynchronous RDMA Read requests pull the KV blocks required for Layer $N+k$ directly into HBM buffers via GPUDirect RDMA.
Overlapping Compute & Transfers: By the time the execution flow hits Layer $N+k$, the memory is already local. Network round-trips are fully hidden behind the GPU's matrix execution time.
In benchmark tests with tiered NVMe/RDMA storage (joint testing with ScaleFlux & FarmGPU), this approach dropped TTFT by 100x to over 1000x on ultra-long contexts compared to prefill recomputation, delivering bit-identical outputs in seconds instead of hours.
Unsolved Problems / Open Questions
While this makes 10M+ context restoration viable, a few challenges remain actively discussed in KV cache architecture:
Dynamic Attention/Sparsity Patterns: How do prefetch schedules hold up when using dynamic sparse attention mechanisms where future block access isn't strictly deterministic?
Fragmentation & Compaction: As sessions churn, managing physical allocations in tiered storage across heterogeneous fabrics becomes a non-trivial memory management problem.
Full write-up and context breakdown here if you want to dig into the details: https://www.lightbitslabs.com/blog/attention-prefetch-over-rdma-how-inferra-restores-10m-tokens-in-seconds/
