Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective
Smaller AI models trained on big-model reasoning may perform worse than before training — a flaw in how the field measures 'knowledge distillation' success.
The Thesis
Chain-of-thought distillation is the practice of training a smaller, cheaper AI model to mimic the step-by-step reasoning of a much larger one. Researchers assumed that when distillation fails, the culprit is a large 'capacity gap' — meaning the student model is simply too small to absorb what the teacher knows. This paper challenges that story by pointing out that most prior experiments never checked whether the student got better compared to its own starting point, only whether it beat other distilled models. When the authors apply that fairer baseline, they find that distillation sometimes makes small models actively worse, and the capacity gap is not the consistent villain it was thought to be. The practical upshot: teams choosing teacher-student pairs for distillation pipelines may be making those choices based on flawed benchmarks.
Catalyst
Large reasoning models — such as OpenAI's o-series and DeepSeek-R1 — have made chain-of-thought distillation a live engineering practice, not just an academic exercise. Companies are actively compressing these models for deployment on smaller hardware, making evaluation methodology a real-world cost question rather than a theoretical one. The proliferation of strong public teacher models in 2024-2025 also created a natural experiment: when you have many candidate teachers of vastly different quality, the capacity gap hypothesis becomes testable in ways it wasn't before.
What's New
Prior distillation research — including influential papers on knowledge distillation benchmarks — compared distilled student models only against each other or against the teacher, never against the un-distilled student's own pre-training baseline. That omission hid cases where distillation was net-negative. This paper introduces a pre-distillation baseline into the evaluation protocol and re-runs experiments across multiple tasks and teacher-student pairings. The authors find the capacity gap's effect is inconsistent across settings, which contradicts the prevailing rule-of-thumb that 'the closer in size, the better the distillation.'
The Counter
This paper's core contribution is methodological, not a new algorithm — and that limits how far its conclusions generalize. The finding that 'capacity gap effects are inconsistent' could simply mean the authors tested a narrow slice of tasks and model families; a different task set might restore the original finding. The proposed fix — always compare against the pre-distillation baseline — sounds obvious in hindsight, and it's unclear why it wasn't already standard practice if the effect were large enough to matter. Distillation practitioners in industry often use continuous fine-tuning pipelines where the 'pre-distillation baseline' is a moving target, making the proposed protocol harder to apply than it appears on paper. Finally, the paper offers practical guidance for teacher-student selection but stops short of proposing a new selection algorithm, so the actionable output for engineers is thinner than the framing suggests.
Longs
- ARM Holdings (ARM) — edge inference chips benefit from better distillation guidance reducing over-engineering
- SoundHound AI (SOUN) — voice AI products depend on small distilled models running locally
- SMLR (Semler Scientific, proxy for small-model efficiency plays) — speculative
- BOTZ (Global X Robotics & AI ETF) — robotics edge AI depends on compact, distilled reasoning models
Shorts
- Vendors selling 'distilled model' products benchmarked only post-distillation — their performance claims may not survive the fairer evaluation protocol proposed here
- Consulting firms and MLOps platforms that have built teacher-selection pipelines around the capacity gap heuristic — those recommendations may need revision
Enablers (Picks & Shovels)
- Hugging Face — hosts the open teacher and student models used in distillation experiments
- EleutherAI LM Evaluation Harness — open benchmarking framework relevant to the evaluation protocol proposed here
- DeepSeek (open weights) — one class of strong teacher models whose public release made this kind of multi-teacher comparison feasible
- PyTorch FSDP / DeepSpeed — distributed training frameworks used to fine-tune student models at scale
Private Watchlist
- Nous Research — open-source fine-tuning and distillation tooling
- Together AI — infrastructure for running and fine-tuning open models at scale
- Predibase — specializes in fine-tuning and deploying smaller language models for enterprise
Resources
The Paper
Chain-of-thought (CoT) distillation transfers reasoning behaviors from a strong teacher to a smaller student, but prior work reports a capacity gap: distillation may fail when the teacher-student capability mismatch is large. We revisit the capacity gap from a practical perspective by re-examining commonly used experimental settings. Notably, we find that CoT distillation often degrades performance compared to the student's pre-distillation baseline, an issue obscured when only post-distillation comparisons are reported. We therefore propose a more realistic evaluation protocol and find that the impact of capacity gap effects does not consistently dominate across tasks and settings, especially when candidate teachers differ substantially in performance. Our results offer practical guidance for selecting teacher-student pairs in CoT distillation.