Asynchronous KV Cache Offloading in vLLM

The py-kvcache connector for vLLM introduces mechanisms to decouple KV cache management from the primary model engine. Historically, retrieving cached prompt prefixes from disk required a synchronous fetch, forcing the engine to stall while waiting for data transfer. The py-kvcache approach enables scheduler-aware I/O by integrating with the vLLM request scheduler to initiate disk reads before the model engine executes its first forward pass. This allows data transfer from NVMe to VRAM to overlap with compute cycles, hiding storage latency that previously halted the inference pipeline.

Intercepting and Preloading Prefixes

The scheduler acts as the gateway for this interaction. When a request enters the queue, the system parses the metadata to extract the block hash and the prompt prefix. Before the engine processes the request, the scheduler queries the external store metadata to verify if the required KV blocks exist on disk. If a match is found, the scheduler triggers an asynchronous I/O thread. This thread performs an asynchronous direct I/O operation to copy the blocks into a pre-reserved staging buffer in GPU memory. To avoid stalling the execution pipeline, the system performs a pointer swap or a shallow copy at the application level to integrate the staging buffer directly into the model's active KV cache memory space. By initiating this read while the request sits in the queue, the system ensures that the necessary data is already available in the addressable cache pool when the scheduler dispatches the request to the GPU executor. This design allows the engine to begin work immediately upon dispatch.

The Memory Trade-off

This approach requires a static memory partition. You must allocate a specific portion of your VRAM as a persistent cache pool. If you size your staging buffers too aggressively to capture more cache hits, you reduce the memory available for active inference and KV storage. This directly limits your concurrent request capacity, forcing a trade-off between cache hit rates and throughput.

Economics of the Break-even Point

External KV caching is an admission decision based on the hardware's arithmetic intensity. On an H100 GPU, the tensor cores regenerate prompt tokens so efficiently that the latency cost of disk streaming often exceeds the cost of recomputation, making the benefit negligible or even negative. The break-even point is a moving target. Consider a 128k prompt context: on mid-range hardware, the compute cost to regenerate those tokens consumes significant cycles, whereas moving that data from a high-speed NVMe drive takes roughly 150 milliseconds. In this case, loading from disk is a clear win. On an H100, the recomputation might happen in 100 milliseconds, rendering the disk-offloading strategy slower than standard recomputation. This mechanism improves the viable window for offloading by reducing effective transfer time, but it cannot overcome the reality that faster compute units decrease the utility of persistent disk caches.

Limitations and Unknowns

• Multi-tenancy Scaling. It is unclear how prefix contention from hundreds of concurrent users affects staging buffer efficiency or whether it leads to excessive thrashing.
• NVMe Variance. We have not tested how NVMe latency variances affect throughput for long-context requests under high contention.
• Static Allocation. The system currently requires a restart to adjust staging buffer sizes, preventing real-time adaptation to evolving traffic patterns.