TCL: Enabling Fast and Efficient Cross-Hardware Tensor Program Optimization via Continual Learning
A new DL compiler framework cuts tuning time 16.8x on CPU — meaning your inference stack just got cheaper to port.
The Thesis
TCL makes cross-hardware tensor program optimization dramatically cheaper by needing only 10% of the training data that incumbents like Tenset-MLP require, while delivering 16.8x faster tuning on CPU and 12.48x on GPU. The core trick is a Mamba-based cost model plus continual knowledge distillation that transfers learned performance intuitions across hardware without retraining from scratch. If this holds outside lab conditions, hardware migration costs for ML inference — currently a major friction point for edge deployment — drop materially.
Catalyst
Mamba's selective state-space architecture, which matured in 2023-2024, is what enables efficient long-range schedule dependency modeling at low parameter counts — that's the specific architectural building block TCL depends on. Simultaneously, hardware fragmentation (AMD GPUs, custom AI ASICs, diverse edge chips) has made the cost of per-hardware tuning increasingly painful for inference-focused teams.
What's New
Prior work like Tenset-MLP required large, hardware-specific offline datasets and couldn't transfer knowledge across platforms without parameter explosion from multi-task learning. TCL's continual distillation sidesteps catastrophic forgetting and data accumulation, which has been the unsolved problem in this stack since at least 2021.
The Counter
The comparison baseline is Tenset-MLP, a 2021-era academic benchmark — not TensorRT, XLA, or vendor-tuned kernels that production teams actually use. Claiming 16.8x speedup over a straw man doesn't tell you much about real deployment friction. More fundamentally, the hardest part of cross-hardware compiler optimization isn't data collection cost — it's handling the long tail of operator shapes and fusion patterns that appear in production models but not in benchmark suites. TCL's RDU Sampler selects 10% of programs by optimizing representativeness, diversity, and uncertainty, but those criteria are measured on the existing dataset distribution, not on the actual workloads a new chip will see. The continual distillation framing is theoretically clean but continual learning papers routinely show strong aggregate metrics while silently forgetting specific task slices — the evaluation here doesn't appear to probe that failure mode rigorously. Until there's an open codebase and third-party reproduction on hardware like MI300X or Trainium, this stays in the 'interesting academic result' bucket.
Longs
- NVDA
- AMD
- INTC
- GOOG
Shorts
- Existing auto-tuning vendors with proprietary offline dataset moats
- Cloud ML compiler SaaS players whose differentiation is dataset scale rather than transfer efficiency
Enablers (Picks & Shovels)
- NVDA
- AMD
- TSMC
- AWS
- Google Cloud TPU team
- Apache TVM open-source ecosystem
Private Watchlist
- OctoML
- Modular
- Tenstorrent
The Paper
Deep learning (DL) compilers rely on cost models and auto-tuning to optimize tensor programs for target hardware. However, existing approaches depend on large offline datasets, incurring high collection costs and offering suboptimal transferability across platforms. In this paper, we introduce TCL, a novel efficient and transferable compiler framework for fast tensor program optimization across diverse hardware platforms to address these challenges. Specifically, TCL is built on three core enablers: (1) the RDU Sampler, a data-efficient active learning strategy that selects only 10% of tensor programs by jointly optimizing Representativeness, Diversity, and Uncertainty, substantially reducing data collection costs while maintaining near-original model accuracy; (2) a new Mamba-based cost model that efficiently captures long-range schedule dependencies while achieving a favorable trade-off between prediction accuracy and computational cost through reduced parameterization and lightweight sequence modeling; and (3) a continuous knowledge distillation framework that effectively and progressively transfers knowledge across multiple hardware platforms while avoiding the parameter explosion and data dependency issues typically caused by traditional multi-task learning. Extensive experiments validate the effectiveness of each individual enabler and the holistic TCL framework. When optimizing a range of mainstream DL models on both CPU and GPU platforms, TCL achieves, on average, 16.8x and 12.48x faster tuning time, and 1.20x and 1.13x lower inference latency, respectively, compared to Tenset-MLP.