FACT: Compositional Kernel Synthesis with a Three-Stage Agentic Workflow
An AI agent now writes GPU kernels by composing mature CUDA libraries, cutting one AI block's runtime by 2.8×.
The Thesis
Writing fast GPU code is a specialist skill that bottlenecks AI deployment. FACT automates this by using a large language model (LLM) to inspect a PyTorch computation graph, identify optimization opportunities, and synthesize kernels using CUTLASS — NVIDIA's production-grade C++ template library for GPU math. The key insight is that instead of asking an AI to write raw GPU code from scratch, FACT anchors generation in a library that already encodes decades of microarchitecture tuning. The result: a 2.79× end-to-end speedup on a realistic AI block (MiniGPT), achieved without a human expert writing a single line of CUDA. The catch is that the approach is currently limited to patterns CUTLASS already supports, and benchmarks are restricted to one GPU and a narrow workload set.
Catalyst
LLMs capable of reasoning over code graphs and generating structured C++ templates have only become reliable enough for this kind of multi-stage synthesis in the past 12–18 months. Simultaneously, KernelBench — a standardized benchmark for GPU kernel generation — gave researchers a shared evaluation harness, making it possible to compare approaches objectively. These two developments together created the conditions for agentic kernel synthesis to move from a curiosity to a testable engineering workflow.
What's New
Earlier LLM-based kernel generators, such as those targeting raw CUDA output, require the model to rediscover low-level GPU optimizations (memory coalescing, tile sizing, warp scheduling) that expert libraries have already solved. Systems like vendor-tuned cuBLAS or hand-written CUTLASS kernels deliver good performance but require human experts to write and maintain them. FACT instead treats CUTLASS as a structured vocabulary — the LLM selects and configures existing, verified building blocks rather than inventing new ones, then auto-tunes parameters by sweeping the CUTLASS configuration space. The authors claim this approach is more reliable and more deployable than raw CUDA generation.
The Counter
The benchmark scope is very narrow: three matrix-multiply variants and one MiniGPT block, all on a single A100. A 2.79× speedup on a carefully chosen fused attention block does not tell us how FACT performs on the messy, irregular operators that dominate real production models. The approach is fundamentally limited to patterns already encoded in CUTLASS — any operator outside that library's catalog forces a fallback, and the paper does not characterize how often that happens in practice. Auto-tuning by sweeping CUTLASS parameters is also not new; tools like NVIDIA's own Nsight and commercial autotuners already do this without requiring an LLM in the loop. Finally, the gains on basic GEMM workloads (6–18% over cuBLAS) are modest and may not justify the engineering overhead of integrating an agentic workflow into a production deployment pipeline.
Longs
- NVDA — CUTLASS is NVIDIA's library; faster kernel adoption deepens A100/H100 ecosystem lock-in
- AMD (AMD) — competitive pressure if similar agentic workflows target ROCm/HIP
- BOTZ (Global X Robotics & AI ETF) — AI inference efficiency is a direct input to robotics deployment costs
- Palantir (PLTR) — AI deployment pipelines that auto-optimize compute are core to AIP platform value
Shorts
- GPU kernel consultancies and freelance CUDA engineers — their core value proposition is automating exactly what FACT targets
- Deep learning compiler projects like Apache TVM or XLA — if agentic CUTLASS synthesis becomes the default path, bespoke compiler stacks lose relevance for common workloads
Enablers (Picks & Shovels)
- CUTLASS (NVIDIA open-source C++ GEMM library) — the foundational vocabulary FACT builds on
- KernelBench — the evaluation framework used to score results; essential for reproducibility
- PyTorch FX graph tracing — the mechanism FACT uses to inspect model computation graphs before optimization
- NVIDIA A100 GPU — current benchmark hardware; generalization to H100 or consumer GPUs is unproven
Private Watchlist
- Modular — builds AI compiler infrastructure directly competing with and complementing this approach
- Voltron Data — GPU-accelerated data processing that benefits from automated kernel tuning
- OctoAI — model serving startup where inference kernel efficiency directly affects unit economics
Resources
The Paper
Deep learning compilers and vendor libraries deliver strong baseline performance but are bounded by finite, engineer-curated catalogs. When these omit needed optimizations, practitioners substitute hand-written CUDA or CUTLASS, demanding expertise in GPU microarchitecture and C++ template metaprogramming. Recent LLM-based agents target kernel generation in raw CUDA, forcing rediscovery of optimizations already encoded in mature libraries. We present FACT (Framework for Agentic CUTLASS Transpilation), a framework that employs a three-stage, agent-driven workflow optimizing PyTorch modules through multi-pattern composition while grounding synthesis in CUTLASS C++. (1) Pattern discovery: an LLM agent inspects the traced graph, matches subgraphs to optimization rules, retrieves vetted examples from an architecture-specific index, and outputs prioritized patterns. (2) Pattern realization: each pattern is implemented as a CUTLASS kernel wrapped in a PyTorch extension, verified, and auto-tuned by sweeping parameters inferred from the CUTLASS hierarchy. (3) Pattern composition: extensions are loaded together into a single composed module for end-to-end benchmarking. We evaluate the workflow using KernelBench's evaluation framework and provided modules on an NVIDIA A100. On Level 1, we apply the workflow to three GEMM workloads (square matrix multiply, batched matrix multiply, and large-$K$ matrix multiply). Auto-tuned CUTLASS kernels improve over PyTorch cuBLAS baseline by $1.06\times$--$1.18\times$. On Level 3 MiniGPT block, composing fused multi-head attention with fused MLP GEMM+GELU yields $2.79\times$ end-to-end speedup. Our work couples agentic graph-level pattern discovery with auto-tuning and a dynamic pattern table, offering a practical path from traced PyTorch to deployable kernels by automating CUTLASS kernel synthesis and auto-tuning.