Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation
A new offline training method cuts the compute cost of distilling large reasoning models by 4x, eliminating the need for a live teacher server.
The Thesis
Training a capable reasoning model today typically requires keeping a large 'teacher' model running continuously to guide a smaller 'student' model — a process called on-policy distillation (OPD). This paper finds that the reason offline shortcuts have historically failed is not a fundamental limitation, but a subtle condition the authors call 'teacher consistency': the same teacher model must be used for both the initial fine-tuning step and the distillation step. Violating that condition introduces a systematic bias that no amount of additional training can correct. Lightning OPD fixes this by precomputing the teacher's outputs once, before training begins, then reusing them throughout — cutting a 30-GPU-hour experiment to roughly 7.5 GPU-hours while reaching 69.9% on AIME 2024 with an 8-billion-parameter model. The catch is that the efficiency gain depends entirely on having a single, consistent teacher; mixing teachers or checkpoints breaks the theoretical guarantee.
Catalyst
The recent proliferation of strong open-weight reasoning models — particularly the Qwen3 series — gives researchers a high-quality, freely available teacher to stay consistent with. Academic labs now regularly attempt post-training runs that were previously gated by cloud inference costs, making a 4x infrastructure reduction concretely meaningful rather than theoretical.
What's New
Standard OPD, as practiced in systems like DeepSeek-R1 and similar pipelines, requires a live teacher inference server running in parallel with training, generating fresh outputs every step. Earlier attempts to sidestep this by caching teacher outputs offline failed because they silently violated teacher consistency — using one teacher for supervised fine-tuning and another (or a different checkpoint) for the distillation signal. Lightning OPD identifies this mismatch as the root cause of degraded performance and redesigns the pipeline so the same teacher that produces the SFT data also provides the cached log-probabilities, closing the gap with online OPD at a fraction of the infrastructure cost.
The Counter
The core claim — that teacher consistency was a previously overlooked but critical condition — is hard to evaluate independently because the paper is simultaneously identifying the problem and proposing the fix. If the condition were truly fundamental, it should be straightforward to reproduce the failure mode on other labs' published OPD pipelines, and the paper does not provide that cross-system validation. The 4x speedup figure is compelling on its face, but it compares against a specific implementation of online OPD rather than against the most optimized production pipelines used by well-resourced teams, which already employ asynchronous inference and caching tricks. More importantly, the benchmark chosen — AIME 2024 mathematical competition problems — is a narrow slice of capability; a model trained entirely with this method might show degraded performance on the open-ended, instruction-following tasks that matter most commercially. Finally, the method's efficiency advantage narrows or disappears if the precomputed rollout set needs to be regenerated frequently to track a changing data distribution, which is common in production post-training pipelines.
Longs
None listed.
Shorts
- Cloud inference API providers (e.g., companies charging per-token for teacher model hosting during training) lose revenue if labs shift to fully offline pipelines that require no live inference calls
- Startups selling 'training-time inference orchestration' tooling lose a key use case if the live teacher server requirement disappears
Enablers (Picks & Shovels)
- Qwen3 open-weight model series (Alibaba) — the specific teacher model used in the paper's experiments
- vLLM — open-source inference engine used to generate and cache teacher log-probabilities efficiently
- Hugging Face Transformers and TRL — standard open-source libraries for the SFT and distillation training loops
Private Watchlist
- Together AI — provides on-demand inference infrastructure that academic labs use for exactly this kind of post-training workload
- Modal Labs — serverless GPU platform used for academic-scale LLM fine-tuning experiments
The Paper
On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, standard OPD requires a live teacher inference server throughout training, resulting in substantial infrastructure overhead. In this work, we investigate whether on-policy distillation can be performed offline. A natural approach is to precompute teacher log-probabilities once over SFT rollouts and reuse them during training. In practice, however, this offline variant fails to reliably match the performance of standard OPD. To understand this discrepancy, we identify a previously overlooked condition that is critical for any OPD pipeline, which we term teacher consistency. This condition requires that the same teacher model be used for both supervised fine-tuning and OPD. We show that violating teacher consistency introduces an irreducible gradient bias, causing both offline and online OPD to converge to a suboptimal fixed point regardless of training duration. Building on this insight, we propose Lightning OPD, an offline on-policy distillation framework that enforces teacher consistency by precomputing teacher log-probabilities over SFT rollouts. This design eliminates the need for a live teacher server entirely. We further show that, under teacher consistency, Lightning OPD shares the same optimum as standard OPD, with bounded gradient discrepancy and an implicit regularization effect that helps prevent policy drift. Extensive experiments on mathematical reasoning and code generation demonstrate that Lightning OPD achieves state-of-the-art performance with significantly improved efficiency. Starting from an SFT-initialized Qwen3-8B-Base model, Lightning OPD reaches 69.9% on AIME 2024 in just 30 GPU hours, achieving a 4.0x speedup over standard OPD and substantially lowering the barrier to entry for academic research on LLM post-training.