Record-Remix-Replay: Hierarchical GPU Kernel Optimization using Evolutionary Search
An LLM-plus-Bayesian framework auto-tunes GPU kernels end-to-end, nearly 10x faster than today's best evolutionary search.
The Thesis
R³ combines LLM-driven code mutation, Bayesian optimization, and record-replay compilation to search the full GPU kernel optimization space — algorithm choice, compiler pass ordering, and launch parameters — simultaneously rather than sequentially. Prior tools force engineers to tune each dimension in isolation, leaving performance on the table; R³ claims to beat those tools on full scientific applications while being ~10x faster to evaluate. If this generalizes beyond scientific computing to ML training kernels, it compresses months of hardware bring-up work into automated pipelines.
Catalyst
LLMs capable of semantically meaningful source-code mutation (not just stochastic perturbation) only became practical at scale in the last 12-18 months, making the 'remix' step viable where prior evolutionary approaches were essentially blind syntax shufflers. Record-replay compilation dramatically cuts per-candidate evaluation cost, which is the historic bottleneck that made full-space search intractable.
What's New
Previous approaches like autotuning frameworks (OpenTuner, KernelTuner) or compiler autoschedulers (Halide, TVM's Ansor) search launch configs or compiler passes in isolation — they don't co-optimize source-level implementation variants. R³ treats all three levels as a joint search problem, which is the actual thing human experts do manually.
The Counter
The '~order of magnitude faster' claim is seductive but needs hard scrutiny: faster than what baseline, on what kernels, measured how? If the comparison is against naive evolutionary search without record-replay, that's a compiler engineering win, not an AI win — and it doesn't validate the LLM mutation component specifically. More importantly, the HPC scientific computing niche already has a shrinking addressable market as ML frameworks increasingly standardize on a small set of heavily hand-tuned kernels (cuBLAS, cuDNN, FlashAttention) that vendors maintain directly. Auto-tuning a bespoke stencil code is a real problem, but it's not the billion-dollar problem; that's ML training throughput, and the paper doesn't demonstrate results there. Finally, LLM-driven code evolution has a reproducibility problem: the search trajectory is non-deterministic and depends on the specific LLM checkpoint used, making peer validation and deployment in regulated HPC environments (national labs, finance) genuinely difficult.
Longs
- NVDA
- AMD
- INTC
Shorts
- Manual kernel engineering consultancies
- Companies selling hand-tuned HPC libraries (e.g., Intel oneAPI/MKL teams whose moat is human tuning expertise)
- TVM/Apache community whose autoscheduler value prop overlaps directly
Enablers (Picks & Shovels)
- NVIDIA CUDA ecosystem
- LLVM/MLIR open-source compiler infrastructure
- Hugging Face (open LLMs used for code mutation)
- major cloud GPU providers: AWS, GCP, Azure
Private Watchlist
- Modular
- Lightmatter
- Cerebras
The Paper
As high-performance computing and AI workloads become increasingly dependent on GPUs, maintaining high performance across rapidly evolving hardware generations has become a major challenge. Developers often spend months tuning scientific applications to fully exploit new architectures, navigating a complex optimization space that spans algorithm design, source implementation, compiler flags and pass sequences, and kernel launch parameters. Existing approaches can effectively search parts of this space in isolation, such as launch configurations or compiler settings, but optimizing across the full space still requires substantial human expertise and iterative manual effort. In this paper, we present Record-Remix-Replay (R^3), a hierarchical optimization framework that combines LLM-driven evolutionary search, Bayesian optimization, and record-replay compilation techniques to efficiently explore GPU kernel optimizations from source-level implementation choices down to compiler pass ordering and runtime configuration. By making candidate evaluation fast and scalable, our approach enables practical end-to-end search over optimization dimensions that are typically treated separately. We show that Record-Remix-Replay can optimize full scientific applications better than traditional approaches over kernel parameters and compiler flags, while also being nearly an order of magnitude faster than modern evolutionary search approaches.