Phoenix
ProductionRefactored I/O Stack for GPU Direct Storage without Phony Buffers
Motivation
Remove the Phony Buffer from the Critical Path
GPU Direct Storage (GDS) enables direct data transfer between GPU memory and NVMe devices using Peer-to-Peer DMA (P2P-DMA), bypassing traditional host bounce buffers. However, current GDS implementations employ a "phony buffer" in host memory to interact with the Linux kernel—a same-sized host memory region that represents the GPU buffer during kernel operations.
This phony buffer creates significant problems:
- Suboptimal Performance: Complex logic for phony buffer management and DMA address replacement adds overhead to the critical I/O path
- Excessive Resource Consumption: Phony buffer requires same-size host memory allocation, wasting RAM and CPU cycles
- Poor Compatibility: Requires customized device drivers and non-POSIX interfaces, complicating deployment and programming
Phoenix eliminates phony buffers entirely by leveraging Linux ZONE_DEVICE to map GPU memory directly into the kernel's page table. This allows the Linux file system stack to interact directly with GPU memory using standard POSIX interfaces, removing the proxy layer and its associated overhead.
Key Design Ideas
ZONE_DEVICE Memory Mapping
Phoenix uses Linux ZONE_DEVICE (kernel 4.3+) to map GPU memory directly into the kernel's page table at system startup, making GPU memory addressable through standard virtual memory operations.
- • GPU memory participates in kernel page model
- • No proxy buffer needed
- • Standard struct page representation
- • Enables POSIX interface compatibility
Standard Linux I/O Interfaces
Phoenix works with unmodified Linux file system, block layer, and native NVMe drivers—no custom kernel modifications required for the core I/O path.
- • Standard POSIX read/write/pread/pwrite
- • Native VFS and page cache integration
- • Unmodified block layer
- • Standard NVMe driver
Asynchronous I/O with io_uring
Phoenix integrates with Linux io_uring for high-performance asynchronous I/O, enabling efficient batching and overlap of I/O operations with GPU computation.
- • Zero-copy asynchronous I/O
- • Batch submission for efficiency
- • Polling and interrupt modes
- • Stream-ordered operations
Application Integration Adapters
Phoenix provides adapters and bindings for seamless integration with AI frameworks and storage systems, minimizing application code changes.
- • libphoenix C/C++ library
- • Python bindings
- • LMCache integration (phxfile adapter)
- • vLLM integration support
Architecture
Evidence
Paper Results
Source: "Phoenix: A Refactored I/O Stack for GPU Direct Storage without Phony Buffers" (SC'25, November 2025)
Key Performance Characteristics
- Eliminates phony buffer allocation and management overhead
- Reduces CPU involvement and host memory consumption
- POSIX interface compatibility enables standard programming patterns
- Compared to state-of-the-art GDS I/O stack in controlled evaluation
Current Support Boundary
Core System
- • ZONE_DEVICE-based GPU memory mapping
- • POSIX interface support (read/write/pread/pwrite)
- • io_uring integration for async I/O
- • Unmodified Linux I/O stack compatibility
- • P2P DMA for direct NVMe-GPU transfers
Integrations
- • libphoenix library (C/C++)
- • Python bindings
- • LMCache adapter (phxfile)
- • Check repository for vLLM/framework status
- • Adapter API for custom integrations
Platform Support
Phoenix is implemented for recently-released GPU and NPU models. Check the GitHub repository README for the current list of tested hardware platforms and deployment requirements.
- →Linux kernel 4.3+ required for ZONE_DEVICE support
- →PCIe P2P DMA capability required (verify GPU/system support)
- →NVMe storage with P2P support recommended for optimal performance
Research
Phoenix: A Refactored I/O Stack for GPU Direct Storage without Phony Buffers
Venue: SC'25 - The International Conference for High Performance Computing, Networking, Storage and Analysis
Date: November 16-21, 2025, St Louis, MO, USA
DOI: 10.1145/3712285.3759862
Authors: Jianqin Yan, Shi Qiu, Yina Lv, Yifan Hu, Hao Chen, Zhirong Shen, Xin Yao, Renhai Chen, Jiwu Shu, Gong Zhang, Yiming Zhang