CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V
A new benchmark uses Civilization V to test AI strategic thinking across hundreds of turns — showing current LLMs have distinct, measurable weaknesses in long-horizon planning.
The Thesis
Most AI benchmarks test what models know, not how they plan. CivBench puts large language models (LLMs) into multiplayer Civilization V — a turn-based strategy game spanning hundreds of decisions against multiple opponents — and measures how well they make strategic choices over time, not just at the end. The core innovation is a progress-based scoring system: instead of waiting for a win or loss signal that might come after 300 turns, CivBench trains a model to estimate each player's probability of winning at every turn, giving a richer feedback signal throughout the game. Across 307 games involving 7 different LLMs, the benchmark reveals that different models have distinct strategic profiles — strengths and blind spots — that a simple win-rate metric would completely hide. The catch: this is a research benchmark, not a deployed product, and its real-world relevance depends on how much multi-turn strategic reasoning in a video game actually predicts performance in higher-stakes domains like logistics, negotiation, or defense planning.
Catalyst
LLMs have only recently become capable enough to play complex strategy games at all, making benchmarks like this newly meaningful rather than trivially one-sided. At the same time, the AI field is shifting from static question-answer evaluations toward agentic setups — systems that take sequences of actions over time — creating demand for benchmarks that measure longitudinal decision quality. Civilization V's modding ecosystem provides an accessible API for programmatic agent control, lowering the engineering barrier enough that academic groups can run hundreds of full games.
What's New
Prior game-based AI benchmarks — such as those built on Chess, Go, StarCraft II, or Diplomacy — either use non-language-model agents, focus on two-player zero-sum games, or rely on sparse win/loss outcomes that provide almost no learning signal during a long game. Some recent LLM game benchmarks use simplified or single-player environments that don't capture competitive, multi-agent dynamics. CivBench introduces turn-level victory probability estimation, validated through three lenses: predictive validity (does the score predict the actual outcome?), construct validity (does it measure what it claims to?), and convergent validity (does it agree with other reasonable measures of progress?) — giving the evaluation metric itself a rigorous foundation that most prior benchmarks skip.
The Counter
Civilization V is a video game with fixed rules, discrete turns, and a known scoring function — none of which resemble the messy, ambiguous, real-time environments where strategic AI actually needs to work. The jump from 'performs well at Civ V' to 'reasons well in defense planning or business strategy' is enormous and completely undemonstrated here. The paper runs 307 games, which sounds like a lot, but across 7 models and multiple agent configurations, the per-cell sample sizes are thin — enough to generate profiles, not enough to make statistically robust claims about capability ordering. The victory-probability estimator is itself a learned model trained on game data; any errors or biases in that estimator propagate directly into every downstream conclusion about which LLM is 'more strategic.' Finally, Civilization V is over a decade old and its strategic patterns are well-represented in LLM training data, raising the uncomfortable question of whether models are doing novel reasoning or pattern-matching against walkthroughs and strategy guides they've already seen.
Longs
- MSFT — Azure AI hosts many LLM deployments that enterprise customers want to evaluate for agentic reasoning
- PLTR — defense and intelligence clients need long-horizon AI decision tools; benchmark methodology directly relevant
- BBAI — smaller defense-AI firm focused on decision intelligence where multi-step reasoning quality matters
- BOTZ (Global X Robotics & AI ETF) — broad exposure to agentic AI infrastructure buildout
Shorts
- Static benchmark providers (e.g., BIG-Bench, MMLU-style leaderboards) — if agentic, longitudinal benchmarks become the standard, single-turn Q&A evaluations lose credibility as proxies for real-world usefulness
- LLM vendors with inflated leaderboard positions — models that score well on static benchmarks but poorly on multi-turn strategy would face uncomfortable public comparisons
Enablers (Picks & Shovels)
- Freeciv and Civilization V modding APIs — open interfaces that make programmatic agent control possible at scale
- Hugging Face's Open LLM Leaderboard ecosystem — established norms for comparative LLM evaluation that CivBench extends into agentic settings
- OpenAI Evals and EleutherAI's lm-evaluation-harness — open evaluation frameworks the community could extend with game-based tasks
- Cloud compute (AWS, GCP) — running 307 multi-hundred-turn games requires significant parallel compute
Private Watchlist
- Adept AI — building LLM agents for multi-step task completion, directly comparable evaluation context
- Cognition AI — agent-focused lab (Devin) where long-horizon planning benchmarks matter for product claims
- Levels.fyi / Scale AI — data and evaluation infrastructure companies that could operationalize game-based benchmarks for enterprise clients
Resources
The Paper
Evaluating strategic decision-making in LLM-based agents requires generative, competitive, and longitudinal environments, yet few benchmarks provide all three, and fewer still offer evaluation signals rich enough for long-horizon, multi-agent play. We introduce CivBench, a benchmark for LLM strategists (i.e., agentic setups) in multiplayer Civilization V. Because terminal win/loss is too sparse a signal in games spanning hundreds of turns and multiple opponents, CivBench trains models on turn-level game state to estimate victory probabilities throughout play, validated through predictive, construct, and convergent validity. Across 307 games with 7 LLMs and multiple CivBench agent conditions, we demonstrate CivBench's potential to estimate strategic capabilities as an unsaturated benchmark, reveal model-specific effects of agentic setup, and outline distinct strategic profiles not visible through outcome-only evaluation.