Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel
A new compiler abstraction cuts LLM inference latency by fusing dynamic GPU kernels — the plumbing war just got interesting.
The Thesis
Event Tensor Compiler (ETC) tackles the last unsolved problem in megakernel fusion: dynamic shapes and data-dependent computation that trip up existing static approaches. If it delivers on state-of-the-art LLM serving latency with reduced warmup overhead, it's a meaningful efficiency wedge for inference-at-scale operators. The real prize is whoever owns the compiler stack owns the margin.
Catalyst
LLM inference has pushed GPU utilization to the point where kernel launch overhead and coarse synchronization are measurable cost lines, not theoretical concerns. Persistent kernel techniques matured enough in 2024-2025 (via FlashAttention, Triton, and vendor fusions) to make a unified dynamic abstraction both necessary and buildable.
What's New
Prior megakernel work (e.g., persistent kernels in FlashAttention, vendor-specific fusions in cuDNN/TensorRT) assumes static shapes and breaks on dynamic, data-dependent graphs common in real LLM serving (speculative decoding, variable-length batches, MoE routing). ETC introduces the Event Tensor abstraction to encode tiled task dependencies that handle both shape and data dynamism natively.
The Counter
Kernel fusion compilers are a graveyard of 'state-of-the-art' academic claims that never survive contact with production inference stacks. NVIDIA has every incentive and resource to solve this internally — and largely already has for static workloads via TensorRT-LLM and FlashInfer. The paper's own framing ('state-of-the-art LLM serving latency') is a claim with no open benchmark, no reproducible code, and no third-party validation cited in the abstract. Dynamic shape handling in compilers is a solved problem in frameworks like XLA and Torch Inductor for most practical cases; the marginal gains from a new abstraction layer may be noise in end-to-end latency when memory bandwidth and model size dominate. Finally, the inference efficiency game is increasingly won at the hardware level (Groq, Cerebras, custom ASIC) not at the software compiler level, which limits ETC's addressable impact even if the paper's claims hold up.
Longs
- NVDA
- AMD
- MSFT
- GOOGL
Shorts
- TensorRT (NVIDIA's own static-fusion stack faces pressure from more dynamic compilers)
- Torch Inductor / TorchDynamo (if ETC proves superior on dynamic workloads)
- Vendor-locked inference runtimes that compete on kernel optimization rather than flexibility
Enablers (Picks & Shovels)
- NVIDIA (CUDA ecosystem, H100/Blackwell target hardware)
- AMD (ROCm, MI300X adoption curve)
- Triton open-source compiler (OpenAI)
- MLCommons (inference benchmarks)
- cloud inference providers (CoreWeave, Lambda Labs)
Private Watchlist
- Modular
- OctoML
- Anyscale
- Baseten
- Groq
The Paper
Modern GPU workloads, especially large language model (LLM) inference, suffer from kernel launch overheads and coarse synchronization that limit inter-kernel parallelism. Recent megakernel techniques fuse multiple operators into a single persistent kernel to eliminate launch gaps and expose inter-kernel parallelism, but struggle to handle dynamic shapes and data-dependent computation in real workloads. We present Event Tensor, a unified compiler abstraction for dynamic megakernels. Event Tensor encodes dependencies between tiled tasks, and enables first-class support for both shape and data-dependent dynamism. Built atop this abstraction, our Event Tensor Compiler (ETC) applies static and dynamic scheduling transformations to generate high-performance persistent kernels. Evaluations show that ETC achieves state-of-the-art LLM serving latency while significantly reducing system warmup overhead.