HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
A new CPU-GPU memory-sharing framework cuts long-context LLM inference costs by up to 3x, but requires specialized CXL memory hardware most deployments don't yet have.
The Thesis
Large language models are increasingly asked to process very long documents — hundreds of thousands of words at once — and the memory structures they use to do that, called KV caches (tables that store intermediate computation results so the model doesn't recompute them), are growing to hundreds of gigabytes. HybridGen proposes splitting that workload between a GPU (the fast graphics processor) and CPU (the general-purpose processor), coordinated through CXL memory — a newer hardware standard that lets CPUs and GPUs share a common, expandable pool of memory over a fast interconnect. The claim is a 1.4x to 3.2x throughput improvement over six competing approaches, while preserving model accuracy. The catch is real: this only works if you have CXL-capable hardware, which is still rare in production data centers and essentially absent from cloud instances.
Catalyst
Two specific changes made this work timely. First, context windows in frontier models have exploded from a few thousand tokens to one million or more, making KV cache memory a genuine bottleneck rather than a theoretical one. Second, CXL 2.0 memory expansion hardware only became commercially available from vendors like Samsung and Micron in 2023–2024, giving researchers a real substrate to build and benchmark on rather than a simulated one.
What's New
Prior KV cache approaches fall into two camps: pruning methods that discard cache entries to fit in GPU memory, losing accuracy, and offloading methods that move cache data to CPU RAM and retrieve it on demand, creating severe latency from data transfers. Both approaches use only one processor at a time for the attention computation itself — the GPU sits idle during CPU offload and vice versa. HybridGen instead runs attention computation in parallel on both CPU and GPU simultaneously, introduces a feedback-driven scheduler to rebalance work as sequence length grows, and uses a semantic-aware cache placement policy to reduce the penalty of accessing memory across NUMA domains — the performance hit that occurs when a processor has to reach across a memory interconnect to get data that 'belongs' to another node.
The Counter
The core bet here is that CXL memory will become widespread in AI inference infrastructure before GPU memory capacity simply scales past the problem. That is not obvious. The H200 already ships with 141GB of HBM3e; Nvidia's Blackwell architecture pushes further. If GPU memory doubles every two years — roughly the historical trend — the KV cache crisis that motivates HybridGen may be a temporary bottleneck, not a permanent one. The benchmarks are also run on a CXL-expanded memory testbed that most readers cannot replicate or purchase today, which limits reproducibility and real-world validation. The paper compares against six baselines, which is commendable, but all of them are software-only approaches — there is no comparison against simply buying more GPU memory or using a multi-GPU setup with NVLink, which is what hyperscalers actually do. Finally, the scheduling and NUMA-aware placement logic add significant system complexity; production teams have historically preferred simpler, well-understood solutions even at some performance cost, because operational reliability matters more than peak throughput.
Longs
- MU (Micron Technology — a primary manufacturer of CXL DRAM modules that this architecture depends on)
- SMCI (Super Micro Computer — builds GPU servers and is actively integrating CXL memory into its rack designs)
Shorts
- Pure KV cache offloading startups (companies building products around CPU-offload inference, such as those extending llama.cpp for server use, face direct competition from a hybrid approach that claims 3x better throughput on the same hardware class)
- Vendors of KV cache compression/pruning products (their accuracy-throughput tradeoff argument weakens if HybridGen can match accuracy at higher throughput without dropping cache entries)
Enablers (Picks & Shovels)
- CXL Consortium standards body (the open standard that makes CPU-GPU shared memory interoperable across vendors)
- vLLM open-source inference engine (the dominant open inference framework this work would need to integrate with to reach production)
- FlashAttention (the optimized attention kernel library; HybridGen's CPU-side attention path would need comparable kernel efficiency to close the gap)
Private Watchlist
- MemVerge (specializes in CXL memory software and tiered memory management for AI workloads)
- Astera Labs (CXL connectivity silicon and memory fabric controllers)
The Paper
As modern LLMs support thousands to millions of tokens, KV caches grow to hundreds of gigabytes, stressing memory capacity and bandwidth. Existing solutions, such as KV cache pruning and offloading, alleviate these but underutilize hardware by relying solely on either GPU or CPU for attention computing, and considering yet limited CPU local memory for KV cache storage. We propose HybridGen, an efficient hybrid attention framework for long-context LLM inference. HybridGen enables CPU-GPU collaborative attention on systems with expanded tiered memory (e.g., CXL memory), addressing three key challenges: (1) multi-dimensional attention dependencies, (2) intensifying CPU-GPU load imbalance with longer sequences, and (3) NUMA penalty of tiered memories. HybridGen tackles these by introducing attention logit parallelism, a feedback-driven scheduler, and semantic-aware KV cache mapping. Experiments with three LLM models with eleven different sizes on three GPU platforms with a CXL-expanded memory show that HybridGen outperforms six state-of-the-art KV cache management methods by 1.41x--3.2x on average while maintaining superior accuracy.