DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
A new storage framework cuts AI response latency by up to 42% on edge devices by bypassing the operating system when pulling AI memory from local SSDs.
The Thesis
Large language models need to store a working memory — called a KV cache, short for key-value cache — that tracks context across a conversation. On edge devices like industrial PCs or on-device AI boxes, this cache often doesn't fit in GPU memory, so it spills onto local storage. DUAL-BLADE proposes a smarter way to manage that spill: it routes cache data through two separate paths depending on how much memory is available, and when memory is tight, it bypasses the operating system's file layer entirely to talk directly to the SSD. The authors report up to 42% lower latency during the answer-generation phase and 2.2x better SSD utilization in their tests. The catch is that this is a systems paper with hardware-specific dependencies — the gains depend on NVMe SSDs (fast local solid-state drives with a direct CPU interface), and real-world deployment at scale hasn't been demonstrated.
Catalyst
Edge AI inference has grown fast enough that devices like NVIDIA Jetson, AMD Ryzen AI PCs, and purpose-built inference boxes are being deployed in factories, hospitals, and vehicles — all with constrained GPU memory. At the same time, modern LLMs have grown large enough that their KV caches routinely exceed what these devices can hold in GPU memory, making storage offloading a practical necessity rather than an edge case. NVMe SSDs have also become cheap and fast enough (sequential read speeds above 7 GB/s) that direct storage access is now a viable low-latency path, not just a last resort.
What's New
Earlier offloading systems like FlexGen and LMDeploy use standard file-system I/O, which routes data through the Linux kernel's page cache — a layer of RAM the OS uses to buffer disk reads. Under memory pressure, this page cache competes with the GPU memory the model actually needs, causing a 'thrashing' effect where the OS repeatedly evicts and reloads data. DUAL-BLADE sidesteps this by mapping KV tensors directly to raw logical block addresses on the SSD — bypassing the file system entirely when memory is tight — and overlapping that I/O with GPU data transfers using a pipeline so neither the GPU nor the SSD sits idle waiting for the other.
The Counter
The paper shows latency improvements in a controlled benchmark, but the evaluation appears to be on a specific hardware setup with particular NVMe drives and memory configurations — it's not yet clear how these gains generalize across the wide zoo of edge hardware actually deployed in the field. Bypassing the kernel page cache also means giving up OS-level memory protection and error handling; that's a real engineering risk in production environments where reliability matters more than microseconds. The 'dual-path' routing logic adds complexity: determining at runtime which path to use requires its own overhead, and the paper doesn't fully characterize cases where the heuristic misfires. Perhaps most importantly, this problem may shrink on its own — hardware vendors like NVIDIA are already integrating larger unified memory pools in next-generation edge chips, which could simply eliminate the need for SSD offloading at the workload sizes studied here. The 2.2x SSD utilization improvement is also a secondary metric that doesn't directly translate to a business outcome without knowing what the baseline SSD cost and lifespan impact are.
Longs
- NVDA — sells Jetson edge AI modules that are direct deployment targets for this kind of optimization
- SMCI (Super Micro Computer) — builds edge and compact AI servers where NVMe offloading would be integrated
- STX (Seagate) / WDC (Western Digital) — NVMe SSD manufacturers who benefit if SSDs become critical AI memory tiers
- MRVL (Marvell Technology) — designs NVMe SSD controllers; faster, smarter storage access raises controller value
- SOXS inverse / short legacy DRAM ETFs — if SSD offloading reduces peak DRAM requirements on edge, some DRAM demand softens
Shorts
- FlexGen (academic project, now partly commercialized via derivatives) — the file-system offloading approach this paper directly benchmarks against and outperforms
- LMDeploy (OpenMMLab / Shanghai AI Lab) — another file-based offloading framework whose kernel-page-cache architecture is the exact design DUAL-BLADE claims to beat
- Cloud-first LLM inference providers — if edge inference becomes fast and memory-efficient enough, enterprises have less reason to send sensitive data to cloud APIs
Enablers (Picks & Shovels)
- Linux io_uring — asynchronous I/O interface that makes kernel-bypass storage practical without custom drivers
- NVIDIA CUDA DMA engines — the GPU-side transfer hardware that DUAL-BLADE pipelines against storage reads
- NVMe over PCIe 4.0/5.0 — the physical interface standard that makes direct LBA access fast enough to be useful
- llama.cpp and similar on-device inference runtimes — open-source frameworks where such offloading could be integrated
- vLLM — popular open-source LLM serving engine whose PagedAttention memory model is the context this work operates within
Private Watchlist
- Hailo Technologies — edge AI inference chip maker; storage-compute integration is a direct product concern
- Cerebras Systems — large-model inference hardware; memory hierarchy design is central to their architecture
- Untether AI — edge inference silicon startup focused on memory-bandwidth efficiency
- Together AI — LLM inference optimization platform that could adopt or compete with such storage techniques
Resources
The Paper
The increasing deployment of Large Language Model (LLM) inference on edge AI systems demands efficient execution under tight memory budgets. A key challenge arises from Key-Value (KV) caches, which often exceed available device memory. Although NVMe-based offloading offers scalable capacity, existing file-based designs rely heavily on the kernel page cache, leading to cache thrashing, unpredictable latency, and high software overhead under memory pressure. We present DUAL-BLADE, a dual-path KV residency framework that dynamically assigns KV tensors to either a page-cache path or an NVMe-direct path based on runtime memory availability. The NVMe-direct path bypasses the filesystem by mapping KV tensors to contiguous logical block address (LBA) regions, enabling low-overhead direct storage access. DUAL-BLADE further incorporates adaptive pipeline parallelism to overlap storage I/O with GPU DMA, improving inference throughput. Our evaluation shows that DUAL-BLADE substantially mitigates I/O bottlenecks, reducing prefill and decode latency by up to 33.1% and 42.4%, respectively, while improving SSD utilization by 2.2x across diverse memory budgets.