Tutti

Research

GPU-Centric KV Cache Storage for Long-Context LLM Serving

View on GitHubDocumentationResearch Paper

Motivation

When KV Cache Becomes I/O Fragmentation

As growing contexts push key-value (KV) cache footprint far beyond GPU HBM and CPU DRAM capacity, KV cache is increasingly offloaded to NVMe SSDs. LLM serving systems manage KV cache as many relatively small, non-contiguous memory blocks (typically 16KB-64KB per block). When these blocks are moved to SSD-backed storage, restoring a long prefix becomes many small, fragmented storage operations.

Unfortunately, restoring KV cache from SSDs suffers from poor I/O performance and incurs significant GPU stalls. This is primarily because:

  • The fragmented GPU memory layout results in small, scattered I/O operations
  • Frequent CPU involvement for each I/O request creates overhead
  • Lack of coordination between KV cache transfers and GPU storage I/O
  • GPU resource contention during cache restoration

Tutti reorganizes this path around GPU-native KV cache objects and GPU-issued storage operations, reducing CPU involvement to coarser-grained orchestration. By providing bulk KV cache transfers and direct GPU access to SSD-stored cache, Tutti saturates NVMe SSD bandwidth and reduces GPU stalls to near zero.

Key Design Ideas

GPU-Native Object Abstraction

Tutti provides a GPU-native object abstraction that enables bulk KV cache transfers and management, allowing direct GPU access to KV cache stored on NVMe SSDs.

  • β€’ GPU file pool for in-memory object management
  • β€’ NVMe file pool based on GPU file systems (GeminiFS)
  • β€’ P2P memory mapping table for direct access
  • β€’ CPU-side interface integrating allocation, indexing, and GPU access

Asynchronous GPU Direct Object I/O

Re-architected GPU storage stack supporting asynchronous direct I/O, where GPU kernels can issue storage operations without CPU intervention for each request.

  • β€’ GPU kernels directly issue NVMe commands
  • β€’ Batched I/O submission reduces overhead
  • β€’ Asynchronous completion handling
  • β€’ Near-zero GPU stalls during cache restoration

Prefix Caching for Agent Workloads

Optimized for AI agent scenarios where long context prefixes are frequently reused across requests, dramatically reducing redundant computation.

  • β€’ Intelligent prefix matching and reuse
  • β€’ High cache hit rates for agent workloads
  • β€’ Efficient storage of common context prefixes
  • β€’ Support for 131:1 input/output token ratios

Multi-Tier Cache Management

Hierarchical caching across GPU HBM, CPU DRAM, and NVMe SSDs, with intelligent placement policies optimized for long-context inference patterns.

  • β€’ HBM for active inference
  • β€’ DRAM for hot cache blocks
  • β€’ NVMe as primary storage tier (100TB+ capacity)
  • β€’ Adaptive promotion/demotion policies

Architecture

01 Β· Application / LLM Serving Workload
KV cache request: object/prefix ID, offset, size
↓
02 Β· Tutti Cache Management Layer
Prefix cache lookup, allocation, tier management
↓
03 Β· GPU Storage Runtime
GPU file pool, NVMe file pool, P2P mapping, async I/O
↓
04 Β· GPU Direct Storage Path
GeminiFS, GPU-issued NVMe commands, direct DMA
↓
05 Β· NVMe SSDs
Direct DMA to/from GPU HBM, 100TB+ capacity

Architecture based on research paper. Actual implementation may vary.

Evidence

Paper Results

Source: "Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving" (arXiv:2605.03375v1, May 2026)

256K+
Supported Context Length (tokens)
94.2%
Cache Hit Rate (SWE-bench Pro traces)
131:1
Input/Output Token Ratio (agent workloads)

Key Performance Characteristics

  • Near-zero GPU stalls through asynchronous direct I/O
  • Saturates NVMe SSD bandwidth for bulk KV cache transfers
  • 100TB+ capacity potential with NVMe SSD tier (vs. limited HBM/DRAM)

Current Support Boundary

Research Architecture

  • β€’ GPU-native object abstraction
  • β€’ Asynchronous direct I/O stack
  • β€’ Multi-tier cache management (HBM/DRAM/NVMe)
  • β€’ Prefix caching for agent workloads

Implementation Status

Tutti is a research project. Check the GitHub repository for current implementation status, supported platforms, and integration guides.

Performance results are from paper evaluation. Production deployments should verify performance in their specific environments.

Important Notes

  • β†’Vendor support and hardware compatibility should be verified in the repository README before deployment
  • β†’Paper results were obtained in controlled research environments with specific configurations
  • β†’Integration with LLM serving frameworks (vLLM, TensorRT-LLM, etc.) status should be checked in repository

Research

Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving

Venue: arXiv preprint

Date: May 2026

arXiv: 2605.03375v1 [cs.OS]

Authors: Shi Qiu, Yifan Hu, Xintao Wang, Wenhao Zhu, Jianqin Yan, Hao Chen, Kaiqiang Xu, Kai Chen, Yiming Zhang

Read Paper (PDF)Research Overview β†’