Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation
Mixing cheap and expensive GPUs for inference just got 2.3x better — heterogeneous clusters may be the cost arbitrage AI infra needs.
The Thesis
Tessera shows that disaggregating LLM inference at the individual GPU kernel level — rather than at the layer or model-shard level — lets operators mix GPU generations and vendors without sacrificing throughput. On five heterogeneous GPU pairings across four model architectures, it delivers up to 2.3x throughput and 1.6x cost efficiency versus existing disaggregation baselines. The kicker: one heterogeneous pair under Tessera can outperform two high-end homogeneous GPUs at lower total cost, which rewrites the capex calculus for inference clusters.
Catalyst
GPU scarcity and stratospheric H100/H200 prices have forced operators to run mixed-generation clusters anyway — Tessera makes that a feature rather than a liability. The availability of PTX-level tooling mature enough to extract precise inter-kernel dependency graphs at scale is also a prerequisite that only recently became tractable.
What's New
Prior disaggregation work (e.g., Splitwise, DistServe) splits workloads at coarse granularities — prefill vs. decode phases or whole transformer layers — and is tightly coupled to specific architectures. Tessera operates at the GPU kernel level, making it architecture-agnostic and able to match fine-grained compute shapes to the hardware best suited for them.
The Counter
The 2.3x throughput number is against 'existing disaggregation methods,' not against the obvious baseline of just buying more H100s or using tensor parallelism on a homogeneous cluster — that comparison is conspicuously absent. Kernel-level disaggregation adds a new failure mode: the scheduler itself becomes a bottleneck and single point of failure in the inference path, and production ML infra teams are already drowning in operational complexity without adding PTX-parsing runtime systems. The whole thesis also depends on heterogeneous GPU pairs being economically available in quantity, but the used/secondary GPU market is thin and cloud providers don't let you mix GPU types within a job in most standard offerings. And even if the throughput gains hold, the engineering cost to integrate Tessera into vLLM, TGI, or TensorRT-LLM — each with their own kernel dispatch logic — is non-trivial. This reads like a compelling systems research result that is years away from a prod deployment.
Longs
- NVDA
- AMD
- INTC
- GOOGL
- AMZN
- MSFT
Shorts
- Pure-play H100/H200 cloud providers charging homogeneous-cluster premiums
- Inference vendors (Fireworks AI, Octo AI) whose pricing assumes homogeneous GPU fleets
- NVIDIA, to the extent that high-end GPU exclusivity in inference is the moat — Tessera partially commoditizes the mix decision
Enablers (Picks & Shovels)
- NVIDIA (PTX toolchain, CUDA runtime)
- AMD (ROCm, alternative GPU supply)
- AWS, GCP, Azure (heterogeneous instance mix in cloud inference fleets)
- CoreWeave (mixed-GPU cluster operator)
- Intel Gaudi (potential beneficiary as a 'cheap' tier in mixed clusters)
Private Watchlist
- Together AI
- Anyscale
- Replicate
- Modal Labs
- Crusoe Energy
The Paper
Disaggregation maps parts of an AI workload to different types of GPUs, offering a path to utilize modern heterogeneous GPU clusters. However, existing solutions operate at a coarse granularity and are tightly coupled to specific model architectures, leaving much room for performance improvement. This paper presents Tessera, the first kernel disaggregation system to improve performance and cost efficiency on heterogeneous GPUs for large model inference. Our key insight is that kernels within a single application exhibit diverse resource demands, making them the most suitable granularity for aligning computation with hardware capabilities. Tessera integrates offline analysis with online adaptation by extracting precise inter-kernel dependencies from PTX to ensure correctness, overlapping communication with computation through a pipelined execution model, and employing workload-aware scheduling with lightweight runtime adaptation. Extensive evaluations across five heterogeneous GPUs and four model architectures, scaling up to 16 GPUs, show that Tessera improves serving throughput and cost efficiency by up to 2.3x and 1.6x, respectively, compared to existing disaggregation methods, while generalizing to model architectures where prior approaches do not apply. Surprisingly, a heterogeneous GPU pair under Tessera can even exceed the throughput of two homogeneous high-end GPUs at a lower cost.