GPUOS: A GPU Operating System Primitive for Transparent Operation Fusion
A GPU runtime layer cuts kernel launch overhead by up to 15x for small AI operations, potentially speeding inference without new hardware.
The Thesis
Modern AI inference is surprisingly bottlenecked not by raw computation but by the housekeeping cost of launching each GPU operation — a cost called kernel launch overhead. GPUOS attacks this by keeping a single long-lived GPU kernel running continuously, feeding it new work through a queue rather than repeatedly starting and stopping. The paper claims up to 15.3x speedup on workloads dominated by many small operations, such as attention mechanisms in transformers or micro-batched inference serving. The catch: gains are specific to small-operation-heavy workloads; large matrix multiplications are already efficient and won't benefit. The approach also requires NVIDIA hardware and adds a new software layer that production teams would need to trust and maintain.
Catalyst
Inference workloads have shifted toward patterns — especially attention in large language models and speculative decoding — that spawn many tiny tensor operations rather than a few large ones, making launch overhead newly dominant. NVIDIA's NVRTC library (a runtime compiler for GPU code) has matured enough to make just-in-time operator injection practical without prohibitive compile latency. PyTorch's TorchDispatch hook, added in recent PyTorch versions, now gives researchers a clean interception point to redirect operations transparently, without forking the framework.
What's New
Earlier approaches to fusing GPU operations — such as CUDA Graph capture, which pre-records a fixed sequence of kernels and replays it — require knowing the exact computation graph in advance and cannot adapt to dynamic shapes or new operators at runtime. Compiler-based fusion tools like XLA and torch.compile also work ahead of time and struggle with dynamic inputs. GPUOS instead uses a persistent kernel architecture with runtime injection: operators are compiled on demand via NVRTC and inserted into a running kernel through device-side function pointer tables, enabling dynamic workloads that neither CUDA Graphs nor static compilers handle well.
The Counter
The benchmark setup here should raise eyebrows. The paper measures speedup on workloads that are, by construction, dominated by kernel launch overhead — a scenario that is real but represents only a slice of production inference. On the workloads that actually consume most GPU time (large matrix multiplications for attention projections, feedforward layers), launch overhead is already negligible and GPUOS offers nothing. CUDA Graphs, which NVIDIA ships and major inference frameworks already use, solve the static-graph version of this problem today without adding a novel synchronization layer. For the dynamic cases GPUOS targets, torch.compile with dynamic shape support is an active NVIDIA and PyTorch investment with large teams behind it. The persistent kernel architecture also carries a real cost: it squats on GPU resources even while idle, which is a serious problem in multi-tenant cloud environments where every idle SM (streaming multiprocessor) represents wasted money. There is no end-to-end benchmark on a real model, no open-source code release mentioned, and no comparison against CUDA Graphs on the workloads where Graphs apply. Until those gaps are filled, the 15x headline should be read as 'ceiling under ideal conditions,' not as production-representative.
Longs
None listed.
Shorts
- Vendors selling dedicated kernel-fusion compilers (e.g., products built around static torch.compile or TensorRT graph optimization) lose differentiation if runtime fusion becomes a lightweight OS-level primitive.
- Inference serving startups whose moat is custom CUDA kernel libraries may find that systematic launch-overhead elimination narrows the gap between their hand-tuned kernels and standard PyTorch.
Enablers (Picks & Shovels)
- NVIDIA NVRTC (runtime GPU compiler, the core enabling technology)
- PyTorch TorchDispatch (interception hook that makes transparent integration possible)
- NVIDIA relocatable device code (RDC) toolchain
Private Watchlist
- Anyscale (Ray Serve inference infrastructure)
- Modal Labs (serverless GPU inference)
- Together AI (inference optimization focus)
The Paper
Modern deep learning workloads often consist of many small tensor operations, especially in inference, attention, and micro-batched training. In these settings, kernel launch overhead can become a major bottleneck, sometimes exceeding the actual computation time. We present GPUOS, a GPU runtime JIT system that reduces launch overhead using a persistent kernel architecture with runtime operator injection. GPUOS runs a single long-lived GPU kernel that continuously processes tasks from a host-managed work queue, eliminating repeated kernel launches. To support diverse operations, GPUOS uses NVIDIA NVRTC to just-in-time compile operators at runtime and inject them into the running kernel through device function pointer tables. This design enables operator updates without restarting the kernel or recompiling the system. GPUOS introduces four key ideas: (1) a persistent worker kernel with atomic task queues, (2) a runtime operator injection mechanism based on NVRTC and relocatable device code, (3) a dual-slot aliasing scheme for safe concurrent operator updates, and (4) transparent PyTorch integration through TorchDispatch that batches micro-operations into unified submissions. The system supports arbitrary tensor shapes, strides, data types, and broadcasting through a generic tensor abstraction. Experiments show that GPUOS achieves up to 15.3x speedup over standard PyTorch on workloads dominated by small operations, including micro-batched inference and attention patterns. GPUOS improves utilization while remaining compatible with the PyTorch ecosystem.