TEMPER: Testing Emotional Perturbation in Quantitative Reasoning
Emotional framing in math prompts cuts AI accuracy by up to 10 points — even when every number stays the same.
The Thesis
Large language models sold on quantitative reasoning fail measurably when the same math problem is wrapped in frustration, urgency, or excitement — not because the numbers change, but because the emotional tone does. This paper introduces TEMPER, a dataset of 5,400 matched pairs that isolates emotional style from mathematical content across three standard benchmarks. The core finding is a 2–10 percentage-point accuracy drop tied specifically to emotional language, not to general paraphrasing. A simple fix — running a 'neutralization' pass that strips emotional framing before the model sees the problem — recovers most of the lost performance. The catch: this mitigation adds a preprocessing step and still leaves a residual gap, and the finding raises harder questions about whether current model evaluations are systematically optimistic.
Catalyst
Frontier models are increasingly being deployed in high-stakes settings — customer support, tutoring, financial advising — where users don't write in clean, neutral prose. At the same time, standard benchmarks like GSM8K and ARC-Challenge have always used emotionally sanitized language, meaning the field has had no principled way to measure this failure mode until now. The controlled emotion-translation framework developed here is what made a clean, causal test possible.
What's New
Prior robustness work on language models focused on adversarial attacks (deliberate prompt manipulation), typos, or semantic paraphrase — changes that alter surface form or meaning. That work left open whether emotionally charged but content-identical prompts cause degradation. TEMPER isolates emotional style as an independent variable by verifying that all quantities and relationships are preserved in each rewrite, then tests eighteen models from 1B to frontier scale, providing the first large-scale causal evidence that emotional framing alone is a source of reasoning failure.
The Counter
The 2–10 percentage-point accuracy drop sounds alarming, but the lower end of that range is within normal prompt-sensitivity noise for these models — not every model in the eighteen tested showed the same magnitude. The 'neutralization' mitigation the paper touts is essentially just prompt engineering, and it's not obvious it generalizes beyond math word problems to open-ended generation tasks. The emotion-translation pipeline itself relies on an LLM to rewrite problems, which introduces its own reliability questions: if the rewriter occasionally subtly changes a quantity or relationship, contamination may not be fully caught by the verification step. More practically, real users don't write in consistently 'emotional' prose — they write in messy, mixed-register language that doesn't map cleanly onto the emotional categories tested here. The benchmark is a useful controlled experiment, but it may overestimate the magnitude of the effect in deployment by studying a stylized extreme rather than the naturalistic middle.
Longs
- CWAN — Clearwater Analytics, sells AI-assisted financial workflow tools where user input is often high-emotion
- DUOL — Duolingo, deploys LLMs in high-stakes tutoring where frustrated learners write emotionally
- AI (C3.ai) — enterprise AI deployments facing real-world noisy user input
- IBEX — customer experience outsourcing, sells LLM-augmented support where emotional prompts are the norm
Shorts
- Model evaluation firms and leaderboards (e.g., LMSYS Chatbot Arena, Scale AI HELM) whose benchmarks use emotionally neutral language — TEMPER shows those scores may overstate real-world performance
- Enterprise LLM vendors who cite GSM8K or ARC scores in sales materials without disclosing this robustness gap
- Automated grading and tutoring startups that assume neutral-benchmark performance translates to classroom use
Enablers (Picks & Shovels)
- GSM8K benchmark (Cobbe et al., OpenAI) — one of the three source datasets used to build TEMPER
- ARC-Challenge benchmark (Allen Institute for AI) — second source dataset
- MultiArith dataset — third source dataset
- Hugging Face Transformers — used to run the eighteen evaluated models
- OpenAI API / Anthropic API — frontier models evaluated in the benchmark
Private Watchlist
- Patronus AI — LLM evaluation and red-teaming startup directly relevant to this failure mode
- Robustly — AI robustness testing tooling
- Galileo — LLM observability and evaluation platform
Resources
The Paper
Large language models are trained and evaluated on quantitative reasoning tasks written in clean, emotionally neutral language. However, real-world queries are often wrapped in frustration, urgency or enthusiasm. Does emotional framing alone degrade reasoning when all numerical content is preserved? To investigate this, a controlled emotion translation framework is developed that rewrites problems into emotional variants while preserving all quantities and relationships. Using this framework, Temper-5400 (5,400 semantically verified emotion--neutral pairs) is constructed across GSM8K, MultiArith, and ARC-Challenge, and evaluated on eighteen models (1B to frontier scale). Two core results emerge: First, emotional framing reduces accuracy by 2-10 percentage points even though all numerical content is preserved. Second, neutralizing emotional variants recovers most of the lost performance, showing both that the degradation is tied to emotional style rather than content corruption and that neutralization can serve as a lightweight inference-time mitigation. Non-emotional paraphrases cause no such degradation, implicating emotional content rather than surface-level changes. Beyond emotion specifically, the benchmark construction procedure provides a general framework for controlled stylistic translation and robustness evaluation.