LMCache Meets Direct Storage: Phoenix Integration Delivers 5.6× TTFT Improvement

Overview

As Agents become mainstream, inference contexts have grown to 1M tokens, with single conversation KV Cache reaching tens of GB. HBM and DRAM capacity is limited and prices continue to rise.

Offloading KV Cache to NVMe SSDs is a more cost-effective choice: 100TB capacity per machine, much lower cost. However, SSDs are the furthest layer from the GPU. High data retrieval latency means GPU idling and utilization drops.

GPU Direct Storage enables storage devices to transfer data directly to GPU memory without host memory involvement.

Phoenix is a full-stack open-source direct storage I/O stack that supports POSIX interfaces, uses io_uring for acceleration, and works across vendors with multiple storage backends. Phoenix provides LMCache with a high-performance direct storage backend, reducing average TTFT by 5.6× under high concurrency and boosting system QPS by over 2× in production trace replay.

💡 About xPU.IO

xPU.io is initiated by Tencent DCPD (Data Computing Platform Department), focusing on xPU direct storage for AI applications, including Phoenix and Tutti (GPU-centric KV cache storage).

KV Cache Capacity and Speed Challenges

Workload Characteristics

Current KV Cache workloads have shifted from chatbot to agent. Agentic workloads feature long-horizon multi-round loops, alternating between inference and tool invocation.

Real trace data from Codex on SWE-bench Pro:

📊 Context at round 30 typically grows to ~80K tokens
📈 Maximum exceeds 180K tokens
🔄 Each round adds only hundreds to thousands of new tokens
💾 Input/Output token ratio: 131:1
✅ Cache hit rate: 94.2%

With prefix caching, prefill is nearly free for cached portions. Cost per round reduces to just the incremental input computation.

Two Core Requirements

📦 Capacity

100K token contexts easily reach tens of GB. Multiple long sessions on one instance quickly fill HBM and DRAM, triggering eviction. Eviction means recomputing the entire prefix from scratch next time.

SSDs are the only medium that can handle this scale: 100TB per machine, long-term retention of session data.

⚡ Speed

The median interval between rounds is only **a few seconds** — that's the entire window for loading. Tens of GB of KV must reach GPU memory in these few seconds.

Whether caching provides benefit depends on whether KV retrieval time is significantly lower than recomputation. If loading is too slow, high hit rates still won't save time.

Current Path: KV Must Transit Through Host Memory

Problem Analysis

Current LMCache file storage solutions treat SSDs as DRAM cache. This tiered structure has characteristics opposite to requirements:

The layer with the most capacity is furthest from GPU with lowest bandwidth

Data traverses PCIe twice with an intermediate copy. The entire process requires CPU participation for initiation, synchronization, and callbacks.

buffer = malloc(len);              // Host memory as staging
cudaMalloc(&devPtr, len);
pread(fd, buffer, len, offset);   // Disk → Host memory
cudaMemcpy(devPtr, buffer, len, H2D); // Host memory → GPU memory

Performance Loss

Measured data (4× PCIe 4.0 NVMe RAID 0):

Metric	Value	Note
🔷 Raw NVMe bandwidth	25 GB/s	iostat observation
🔶 End-to-end KV loading	10.2 GB/s	Measured performance
📉 Bandwidth loss	<40%	Less than 40% of raw disk

Codex SWE-bench replay test (20,230 requests):

Average prompt length: ~68K tokens
TTFT average: 1101 ms
- SSD loading time: ~600 ms (blue portion)
- GPU prefill: 313 ms
Host memory staging causes additional latency

Phoenix: Direct Storage Without Phony Buffers

Core Concept

Phoenix is an open-source GPU direct storage middleware, initially published at SC’25.

It solves the core problem: traditionally, data from storage devices (like NVMe SSDs) must pass through CPU and memory before copying to GPU memory — a long path with high overhead.

Phoenix bypasses the CPU, delivering storage data directly to GPU memory, significantly improving data loading speed and freeing CPU resources.

Technical Advantages

✅

Vendor-Agnostic

Built on standard OS I/O stack, no proprietary drivers required

✅

Compatible with Existing Systems

No need to modify applications or filesystems

✅

Low Deployment Barrier

No umount/mount operations needed, fully leverages NVMe bandwidth via io_uring/libaio

✅

Cross-Vendor Support

Validated on NVIDIA, MetaX, and other vendors

LMCache Integration with Phoenix

GDS L1 Framework

LMCache natively supports direct storage through an open GDS L1 framework. It uses slab files on NVMe as L1 storage, comparable to CPU pinned DRAM and Device-DAX.

The GDS L1 framework consists of two layers:

🔷 DMA Driver Layer

✦ Core: process-global GDSContext
✦ Drives DMA between GPU and slab file
✦ Maintains small staging buffer
✦ Uses streams to ensure operation ordering

🔷 Backend Abstraction Layer

✦ Provides unified GDS Backend abstraction
✦ Allows different direct storage solutions to integrate
✦ GDSContext is backend-agnostic
✦ Only interacts with standard APIs

Stream Ordered I/O

Phoenix implements Stream Ordered I/O support. Using accelerator runtime’s LaunchHostFunc primitive, Phoenix wraps host-initiated DMA read/write as a host callback attached to the caller’s stream.

Ordering guarantees:

📍 Kernels queued after a DMA Read will see the data DMA brought back
📍 Gather kernels queued before a DMA Write will have completed; DMA reads stable data
📍 No extra synchronization primitives needed; DMAs on the same stream execute in submission order

Integration Method

Thanks to GDS L1’s openness, integrating Phoenix into LMCache is natural and lightweight.

Phoenix provides a frozen-ABI userspace library libphxfile specifically for LMCache. Based on this library, it implements the interfaces required by the backend abstraction layer, enabling GDSContext to seamlessly perform DMA between GPU and NVMe through Phoenix.

Phoenix Backend Architecture

Layered Structure

Application & Framework
  vLLM / LMCache / Custom Applications
         ↓
Application Adapters
  phxloader / phxcache / phxfile
         ↓
Userspace libphoenix
  Device discovery, memory registration, address resolution
  Sync/Batch/Async I/O ↔ phoenixfs
         ↓
Linux Filesystem & Block Layer
  pread/pwrite or io_uring
  → NVMe/RDMA DMA

Key Features

🔌 Adapter Layer

Connects AI applications. Phoenix delivers capabilities through adapters:

▸ libphxfile: Pure C library for LMCache KV Cache offloading
▸ phxloader: pybind11 extension for vLLM model weight loading

The adapter layer is also a decoupling point: Phoenix internals can evolve without application awareness

📂 I/O Layer

Reuses Linux native I/O stack:

✓ GPU memory becomes I/O-addressable in Linux
✓ Directly uses pread/pwrite, io_uring, thread pools
✓ Batch I/O interface based on io_uring: submit N requests in one syscall
✓ 4× PCIe 4.0 NVMe fully saturates hardware bandwidth at 512 concurrency

🔧 Kernel Layer

Makes GPU memory kernel-recognizable:

◆ Leverages ZONE_DEVICE mechanism (Linux 4.3+)
◆ Creates struct page for GPU memory
◆ GPU pointers become legitimate addresses, no phony buffers
◆ I/O goes directly through standard filesystem, block layer, and native NVMe driver

🌐 Compatibility

Universal hub between storage and GPU:

★ Kernel-side abstracts GPU memory P2P mapping
★ Userspace abstracts device discovery and memory copy primitives
★ Adding new accelerators requires only two implementation files
★ Supports ext4/xfs/NFS/BeeGFS/Lustre and other filesystems

Performance Results

Test Configuration

Test Environment:

🖥️ Hardware: 8×H20 (96GB HBM each) + 4× NVMe Gen4 RAID 0
🤖 Model: TP=8, 256K context
📊 Trace: Production trace (11,596 requests)
👥 Users: 6–476 requests per user, avg length 70K, max 256K
💾 KV Cache capacity: ~3.5TB

Three Configuration Comparison

🟠 DRAM Only

1TB DRAM
LRU eviction
Cache miss recomputes

🔵 SSD Staging

25TB SSD + 400GB DRAM
Via host memory
CPU participation required

🔴 Phoenix

25TB SSD
GPU direct storage
No host memory

Key Metrics

🎯 Performance Improvement

5.6×

TTFT Improvement

vs DRAM Only (high concurrency)

2.9×

QPS Boost

vs SSD Staging

Service Capacity Comparison (TTFT 5s threshold):

🟠 DRAM Only: 7 concurrent
🔵 SSD Staging: 13 concurrent
🔴 Phoenix: 37 concurrent

Performance Analysis

Why is Phoenix faster?

Eliminates host memory staging → Direct reduction in per-load latency
More KV fed to GPU in same time → Higher throughput
Scheduling optimizations work again → KV loading overlaps with inference
Loading faster than computation → Bubbles compressed into compute gaps
Higher concurrency, flatter curve → Scaling advantages evident

Using LMCache+Phoenix

Quick Start (Three Steps)

# ① Build and install Phoenix (kernel module + libphoenix)
git clone https://github.com/xPU-IO/Phoenix.git && cd Phoenix
mkdir -p build && cd build && cmake .. && make -j && sudo make install
nvidia-smi  # Ensure vendor driver is loaded
sudo make insmod  # Load phoenixfs kernel module

# ② Build and install LMCache adapter (libphxfile.so)
bash adapters/LMCache/phxfile/install.sh

# ③ Start LMCache with backend specification
LMCache server ... \
  --gds-l1-backend phx \
  --gds-l1-path /mnt/nvme/LMCache_gds

Notes:

--gds-l1-path points to any writable directory; slab is a regular file there
LMCache only interacts with libphxfile.so; libphoenix is transparent
phxFile* ABI is frozen; Phoenix internal evolution is absorbed at shim layer
Phoenix currently supports Linux kernels 5.0+

Conclusion

💫 Summary

📦 KV Cache needs greater capacity; SSDs are currently the only cost-effective choice

⚡ Capacity solved, speed not yet: data from disk to GPU memory must stage through host memory, reducing bandwidth and engaging CPU

🔧 LMCache provides an open, flexible KV Cache management framework, allowing different storage backends to integrate easily into inference systems

🚀 Phoenix straightens this path, enabling storage to write directly to GPU memory. With faster loading, L1 no longer requires expensive DRAM

💰 Using SSD as L1: greater capacity, lower cost, more concurrency on same hardware with same latency requirements

References

📄 Phoenix: A Refactored I/O Stack for GPU Direct Storage without Phony Buffers (SC’25)
📊 Codex / SWE-bench Pro agentic trace dataset
🌐 xPU.IO Community
💻 Phoenix Project
🔗 LMCache PR #4673
📖 Build and Installation Guide
🔧 Vendor Porting Guide
🏗️ Architecture and Data Path
🔌 LMCache Adapter
📚 libphxfile (Frozen ABI)

🎉

Thanks for reading!

Join the xPU.IO community to explore the future of AI infrastructure