SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions
A new data-curation method called SUPERNOVA extends reinforcement learning training to everyday reasoning tasks — not just math — improving benchmark scores by up to 53%.
The Thesis
Most recent advances in training AI reasoning have focused on domains where answers are easy to check, like math equations and code that either runs or doesn't. SUPERNOVA proposes a way to extend that same training approach — reinforcement learning with verifiable rewards — to messier reasoning tasks like understanding cause and effect, temporal sequences, and logical puzzles. The key insight is that existing human-annotated instruction datasets already contain the raw material needed; the hard part is picking and mixing the right tasks. If the technique holds up beyond benchmarks, it could close a real gap in how AI models handle the kind of reasoning that shows up in legal, medical, and scientific work.
Catalyst
Reinforcement learning with verifiable rewards became practical for LLMs around 2024-2025, driven by models like DeepSeek-R1 and o1. The missing ingredient was curated training data for non-formal reasoning — the authors address this directly by mining existing instruction-tuning datasets rather than building new ones from scratch. The availability of open, high-quality instruction datasets (such as those used in Flan and SuperNatural Instructions) provides the raw material that was not systematically exploited for this purpose before.
What's New
Prior RLVR work — exemplified by OpenAI's o1 and DeepSeek-R1 — focused almost exclusively on math and coding tasks, where correctness is automatically checkable. Those systems leave a large swath of real-world reasoning (causal, temporal, analogical) largely unaddressed because there was no principled way to construct verifiable training signals for it. SUPERNOVA reframes existing instruction-tuning datasets as a source of verifiable rewards, then runs over 100 controlled experiments to identify which task types, mixing ratios, and data augmentations actually transfer to held-out reasoning benchmarks.
The Counter
The benchmark improvements are real but narrow: BBEH, Zebralogic, and MMLU-Pro are still artificial tests, not real-world deployments. The paper runs 100+ ablations, which sounds rigorous, but without a clear held-out test set defined before the experiments began, there's a risk that task selection is overfit to these specific benchmarks. The 52.8% relative gain on BBEH sounds large, but relative improvements from a low base can be misleading — the paper doesn't prominently report absolute accuracy levels throughout. More importantly, the framework depends on instruction-tuning datasets that themselves have known quality and coverage issues; garbage-in dynamics may limit how far this approach scales. Finally, frontier lab competitors (OpenAI, Anthropic, Google DeepMind) have internal data pipelines orders of magnitude larger than anything built on public datasets, so the practical advantage for anyone other than academic or mid-tier commercial developers is unclear.
Longs
- GOOG/Alphabet — DeepMind and Google Brain actively develop reasoning-focused LLMs that would benefit from improved data-curation methods
- META — open-source Llama development pipeline could incorporate SUPERNOVA-style curation for next-generation models
- SOUN (SoundHound AI) — spoken-language reasoning products depend on causal and temporal inference improvements
- BBAI (BigBear.ai) — defense AI analytics products hinge on non-mathematical reasoning quality
- BOTZ (Global X Robotics & AI ETF) — broad exposure to applied AI systems that require general reasoning
Shorts
- Proprietary instruction-dataset vendors whose moat is curated reasoning data — SUPERNOVA shows public datasets can be repurposed effectively
- Companies selling specialized 'reasoning fine-tuning' services without a principled task-selection methodology — the paper shows naive task mixing underperforms targeted selection
Enablers (Picks & Shovels)
- Hugging Face datasets hub — houses the instruction-tuning datasets that SUPERNOVA mines for training signal
- Qwen model family (Alibaba) — used as a baseline, implying the open-weight ecosystem makes this kind of ablation research feasible
- BBEH benchmark (Big-Bench Extra Hard) — the evaluation framework that makes cross-paper progress measurement possible
- GitHub (https://github.com/asuvarna31/supernova) — open code and data release enables community replication and extension
Private Watchlist
- Cohere — enterprise LLM provider whose instruction-following products would benefit from stronger general reasoning
- Mistral AI — open-weight model developer that competes directly on reasoning benchmarks
- Scale AI — data labeling infrastructure is a direct upstream dependency for any instruction-tuning curation pipeline
- Imbue (formerly Generally Intelligent) — focused specifically on reasoning and agency in AI systems
Resources
The Paper
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved large language model (LLM) reasoning in formal domains such as mathematics and code. Despite these advancements, LLMs still struggle with general reasoning tasks requiring capabilities such as causal inference and temporal understanding. Extending RLVR to general reasoning is fundamentally constrained by the lack of high-quality, verifiable training data that spans diverse reasoning skills. To address this challenge, we propose SUPERNOVA, a data curation framework for RLVR aimed at enhancing general reasoning. Our key insight is that instruction-tuning datasets containing expert-annotated ground-truth encode rich reasoning patterns that can be systematically adapted for RLVR. To study this, we conduct 100+ controlled RL experiments to analyze how data design choices impact downstream reasoning performance. In particular, we investigate three key factors: (i) source task selection, (ii) task mixing strategies, and (iii) synthetic interventions for improving data quality. Our analysis reveals that source task selection is non-trivial and has a significant impact on downstream reasoning performance. Moreover, selecting tasks based on their performance for individual target tasks outperforms strategies based on overall average performance. Finally, models trained on SUPERNOVA outperform strong baselines (e.g., Qwen3.5) on challenging reasoning benchmarks including BBEH, Zebralogic, and MMLU-Pro. In particular, training on SUPERNOVA yields relative improvements of up to 52.8\% on BBEH across model sizes, demonstrating the effectiveness of principled data curation for RLVR. Our findings provide practical insights for curating human-annotated resources to extend RLVR to general reasoning. The code and data is available at https://github.com/asuvarna31/supernova.