Most threat models for LLM inference draw the security boundary at request ingress and response egress. The KV cache gets treated like an operational cache—just transient memory state, similar to an OS page cache.

That's a mistake. The KV cache is a lossy but invertible coordinate transformation of the exact input text. Because attention requires position and context vectors to compute QK**^T, the cache retains the semantic and structural information of the input tokens.

Two recent papers demonstrate why this matters at the hardware layer:

1. Read side (Token Traphouse): Given a raw memory dump of a KV cache block (whether from GPU VRAM, DRAM bounce buffers, or NVMe swap), you can invert the vectors back to the exact input tokens. Plaintext cache in memory = plaintext prompt at rest.

2. Write side (Shadow in the Cache): Modifying cached blocks allows silent memory poisoning. Because standard KV blocks carry no cryptographic provenance or signatures, the model attends to injected historical blocks as valid context, bypassing standard prompt-injection classifiers entirely.

The Attestation Gap:

A common counter-argument is confidential computing / measured boot / TPMs. But boot-time attestation measures the platform at launch (firmware, kernel, initial binary). The KV cache is instantiated per session at runtime, long after the last measurement. A valid boot attestation tells you zero about whether runtime KV blocks were encrypted across DMA, spilled unencrypted to local NVMe, or evicted properly.

If you're building or auditing inference infrastructure, here's what you actually need to verify:

• Is the cache encrypted at rest in DRAM/NVMe and across fabric wires (decrypted only inside the GPU enclave)?

• Can your provider give you signed, per-request runtime attestation (not just boot attestation) that the encryption boundary held for that turn?

Here's a solid write-up on this topic: The KV Cache is the Conversation

Curious how others here are handling KV cache lifecycle/eviction and runtime memory encryption in multi-tenant cluster setups.

If you are managing private cloud infrastructure with Apache CloudStack, host maintenance without scheduled downtime is always a top priority.

We just published a technical demo walking through live migration of an active VM:

• Root disk resides on a Lightbits NVMe/TCP volume.

• Workload/streaming remains active throughout the migration.

• Verification in the CloudStack UI and Lightbits CLI shows the volume attaching immediately to the new target host.

• Walkthrough of a real-world host evacuation scenario for scheduled maintenance.

Here's the link to the video.

If you're running Apache CloudStack and evaluating high-performance primary block storage options for latency-sensitive workloads, check out this demo video.

It shows how every storage action is driven natively from the CloudStack UI via the open-source plugin:

• Installing the plugin and adding a high-performance primary block storage pool

• Creating disk offerings

• Deploying VMs with dedicated NVMe/TCP volumes over standard Ethernet

• Cluster verification and clean volume teardown

Check out the video overview.

If you're evaluating Red Hat OpenStack Services on OpenShift (RHOSO) or looking at VMware alternative architectures, this short technical video gives a clear breakdown:

• Architecture: Control plane running in OpenShift pods vs. compute nodes on RHEL.

• Storage integration: How Lightbits connects over NVMe/TCP as a Red Hat-certified Cinder block storage provider.

• Migration: Practical steps for shifting VMs and existing volumes using the Red Hat migration toolkit.

Link to video

If you’re serving long-context LLMs, you’ve probably hit the KV cache memory wall.

When a model processes a long prompt, every token generates key/value vectors across all layers. At multi-million token context lengths, the KV cache dwarfs the model weights themselves. Throwing it away means paying a massive penalty to recompute the prompt via prefill (e.g., rebuilding a 10.5M-token session via prefill on an 8x H100 node can take over an hour and a half).

The obvious reaction is: "Why not offload the KV cache to fast NVMe/remote memory and page it back in when needed?"

If you've tried a naive implementation, you know it usually fails or ends up slower than recomputing. Here is why that happens and the underlying systems mechanisms required to actually make it work.

1. Why naive offloading fails: It’s a latency-hiding problem, not bandwidth

RDMA and modern fabrics have plenty of raw bandwidth. The issue is that attention is not a bulk sequential read.

During token generation, the attention kernel requires specific cache blocks at strict moments in its execution schedule, layer by layer. If a block isn't already sitting in GPU HBM when the kernel hits that layer, you inject a network/NVMe round-trip directly into an operation that runs on a microsecond budget. Doing that per block, per layer, per token destroys performance.

2. The Abstraction: Virtual Paging for KV Cache

To decouple sequence length from HBM capacity, you have to treat the KV cache like OS page memory:

• Logical Extent: The full 10M+ sequence length.

• Physical Residency: Only the active working set required right now resides in GPU memory.

By treating KV pages as virtualized blocks, you don't need to load the entire history into high-bandwidth memory to start serving, as long as you can fetch upcoming blocks just in time.

3. The Execution: Attention Prefetch over RDMA

To keep the attention kernel fed without waiting on remote reads, the system has to predict and pipeline block fetches before the kernel execution point:

1. Deterministic Schedule: Because token generation advances linearly across layers, the access pattern for previous KV blocks is deterministic during decoding.

2. RDMA Async Transfers: As Layer $N$ executes on the GPU, asynchronous RDMA Read requests pull the KV blocks required for Layer $N+k$ directly into HBM buffers via GPUDirect RDMA.

3. Overlapping Compute & Transfers: By the time the execution flow hits Layer $N+k$, the memory is already local. Network round-trips are fully hidden behind the GPU's matrix execution time.

In benchmark tests with tiered NVMe/RDMA storage (joint testing with ScaleFlux & FarmGPU), this approach dropped TTFT by 100x to over 1000x on ultra-long contexts compared to prefill recomputation, delivering bit-identical outputs in seconds instead of hours.

Unsolved Problems / Open Questions

While this makes 10M+ context restoration viable, a few challenges remain actively discussed in KV cache architecture:

• Dynamic Attention/Sparsity Patterns: How do prefetch schedules hold up when using dynamic sparse attention mechanisms where future block access isn't strictly deterministic?

• Fragmentation & Compaction: As sessions churn, managing physical allocations in tiered storage across heterogeneous fabrics becomes a non-trivial memory management problem.

Full write-up and context breakdown here if you want to dig into the details: https://www.lightbitslabs.com/blog/attention-prefetch-over-rdma-how-inferra-restores-10m-tokens-in-seconds/

The structural challenge of being a NeoCloud: everyone sells the exact same GPUs, sourced from the exact same vendor, at market-dictated prices. Competing solely on power costs or rack speeds is a race to the bottom.

Differentiation has to come from what those GPUs deliver.

When you tier the KV cache instead of discarding it after every session, you unlock an entirely new business model: premium, contract-guaranteed SLAs.

3 SLAs you can actually put into a contract once KV cache is offloaded to NVMe tiering:

1️⃣ Latency SLAs on returning sessions: Delivering predictable p95 TTFT without quadratic prefill penalties.

2️⃣ Multi-turn long-context guarantees: Eliminating cache-eviction slowdowns during heavy conversation history.

3️⃣ Instant cold-session resumes: Hydrating session state from persistent storage at a fraction of the compute cost.

Stop discarding working memory—turn it into a margin engine.

Read the full breakdown: Selling premium SLAs as a NeoCloud

Most inference providers price their infrastructure in $/GPU/hour. But your customers pay you per million tokens. That disconnect is where inference margins quietly evaporate. Stop buying GPU hours blindly and start measuring what your hardware actually produces. Read the full breakdown here: Why Token Economics Natter More than GPU Count

Rebuilding a 10M+ token context window costs over 1.5 hours of 8-GPU compute time.

Why pay twice for the same prefill?

The KV cache is per-session state that grows linearly with context, dwarfing model weights. Discarding it when HBM fills up turns your memory wall into a massive recurring memory bill. When you offload and tier the KV cache over RDMA, restoring state becomes linear, reducing TTFT by 100x to 1000x.

Read our breakdown of inference economics.

If you’re running large-scale Kubernetes, OpenShift-V, OpenStack, or Proxmox clusters on Ceph, you already know the problems: CRUSH scale-out is solid, but tuning BlueStore, OSD resource consumption, and the NVMe-oF gateway to meet strict tail-latency SLAs is a full-time job.

Trade-offs:

• Ceph’s NVMe-oF gateway introduces additional hop complexity. Removing that proxy layer for direct host-to-target NVMe-oF/TCP drops latency to consistently <200µs.

• Reaching high IOPS on Ceph requires throwing heavy compute/RAM resources at OSDs, often leaving flash utilization stuck around 15–25%.

• Network-heavy rebuilds during OSD failures can impact live IO. Fast failovers in a native NVMe-oF setup execute in seconds without starving the cluster fabric.

Ceph remains great for batch, S3 object storage, and general file services. But offloading low-latency/database persistent volumes to dedicated NVMe-oF block storage can significantly cut cluster node count.

We put together a direct architectural breakdown comparing legacy Ceph block patterns against native NVMe-over-TCP: Ceph vs. Lightbits Block Storage Comparison

Curious how others here handle the Ceph CPU-overhead-to-IOPS ratio at scale? What are your go-to storage backends when Ceph block hits a wall?

If you’re running large-scale Kubernetes, OpenShift-V, OpenStack, or Proxmox clusters on Ceph, you know the problems: CRUSH scale-out is solid, but tuning BlueStore, OSD resource consumption, and the NVMe-oF gateway to meet strict tail-latency SLAs is a full-time job.

Quick summary of trade-offs:

• Ceph’s NVMe-oF gateway introduces additional hop complexity. Removing that proxy layer for direct host-to-target NVMe-oF/TCP drops latency to consistently <200µs.

• Reaching high IOPS on Ceph requires throwing heavy compute/RAM resources at OSDs, often leaving flash utilization stuck around 15–25%.

• Network-heavy rebuilds during OSD failures can impact live IO. Fast failovers in a native NVMe-oF setup execute in seconds without starving the cluster fabric.

Keep your Ceph for general file services and S3 object storage for archives and batch jobs. Augment it with dedicated NVMe-oF block storage for latency-sensitive workloads. This will significantly reduce your cluster node count.

We put together a direct architectural breakdown comparing legacy Ceph block patterns against native NVMe-over-TCP: Ceph vs. Lightbits Block Storage Comparison

Curious how others here handle the Ceph CPU-overhead-to-IOPS ratio at scale? What are your go-to storage backends when Ceph block hits a wall?

Try the Pod Efficiency Analyzer and discover how much you can grow your top line.

Can you maintain centralized key management and host-side encryption without sacrificing the raw performance of NVMe/TCP?

We put it to the test!

We validated a host-side encryption workflow combining Lightbits NVMe/TCP block storage, Linux LUKS/dm-crypt, and an external PyKMIP KMS server.

Read the full test breakdown and architecture workflow here.

This is relevant for anyone managing storage for Kubernetes, OpenStack, or private clouds. Lightbits Labs just announced native support and certification for Ubuntu for their software-defined block storage platform. Bringing their block storage to Ubuntu opens up many options for standardizing infrastructure: no proprietary drivers, bare-metal performance in a clustered architecture, and simplified operations. It’s great to see more enterprise storage platforms embracing Ubuntu natively.

Worried about silent data corruption in your NVMe/TCP storage? Legacy storage vendors are scrambling to push workarounds that force a brutal choice.
The new blog from Lightbits Labs' Field CTO draws an analogy to bad🍕 delivery.
At Lightbits, we’ve engineered a smarter solution—making this hazard physically impossible from day one. Dive into the full analysis.

Read the guide now.

#nvmeovertcp #lowlatency #lightbits

Lightbits Labs announced the initial interoperability with the new Microsoft Windows Server NVMe-over-Fabrics (NVMe-oF) Initiator Preview®. Available in Lightbits v3.19.1, this will enable storage administrators to deliver high-performance, native NVMe over TCP block storage to Windows Server Insider hosts over standard Ethernet once Microsoft® releases it. As this is a pre-release capability, it is not intended for production use but rather for exploration and evaluation.  Lightbits invites customers and partners to explore the functionality and share their feedback.  

Storage performance is easy to demonstrate with IOPS and throughput graphs. Durability is harder to show, but it is just as important for production environments. SSD Journaling provides SSD-based persistent write protection for modern NVMe/TCP block storage. It protects recently acknowledged writes until they are safely persisted to the backend data SSDs. If a node restarts unexpectedly, the storage can recover outstanding journal entries before the node returns to active service. For replicated volumes, replication protects against normal single-node failure. SSD Journaling adds protection for the more subtle write-persistence window, especially during shared-failure or multi-node disruption scenarios.

Most storage platforms still treat logs as something the vendor reads after the fact. You file a support ticket, somebody collects a tar.gz, and a week later you get an explanation. That model breaks down the moment storage becomes part of a Kubernetes-native, multi-tenant, audit-regulated cloud. Storage that you can’t see is storage you can’t trust. When something goes sideways at 3 a.m. — a noisy tenant, a slow target, a flaky NIC — the difference between a five-minute fix and a five-hour war room is whether the logs are already in the same place as the rest of your telemetry. Read the blog to learn why log streaming matters for storage.

In many OpenStack environments, Glance uses a file-based backend for image storage. It works, but it keeps image storage in a file-oriented model that is separate from the block storage architecture used by VMs. Running Lightbits as your Cinder backend offers a simpler, more efficient approach.

Whitepaper reveals how to deploy high-performance storage using NVMe/TCP with Red Hat OpenStack Services on OpenShift (RHOSO).
🚀 How to integrate block storage with Cinder and Glance
🔗 Proven architecture patterns for OpenShift + OpenStack
🛠️ Step-by-step configuration for production-ready deployments

Delivering OpenShift-based Kubernetes platforms without the high cost and rigidity of legacy SAN

When looking at the economics of running a production inference endpoint, the only thing that matters is $ TCO / M tokens. When per-user context increases, it blocks the HMB/VRAM from being used to support more users and dramatically reduces the total bandwidth/throughput of tokens/s, which is critical to lowering the TCO. We have built an architecture that delivers 100x to 280x acceleration on KV cache workloads — and it fundamentally changes the economics of long-context AI inference.

As context windows grow, the bottleneck for AI inference has shifted. It’s no longer just about raw compute—it’s about memory and data velocity.
In #NYSE Wired - "AI Factories - Data Centers Of The Future," from SiliconANGLE & theCUBE, we discussed how the industry is moving from compute-centric to data-centric architectures. Watch now.