Can Vision Language Models Judge Action Quality? An Empirical Evaluation
State-of-the-art vision-language models can barely beat random chance at judging movement quality, exposing a critical gap for sports-tech and physical therapy AI.
The Thesis
Vision-language models (VLMs) — AI systems that reason jointly over video and text — are increasingly pitched as tools for automated coaching, sports judging, and physical rehabilitation feedback. This paper is a sobering reality check: even top models like Gemini 2.5 Pro, Qwen3-VL, and InternVL3.5 perform only marginally better than random guessing when asked to assess whether a gymnast's dive or an athlete's squat was executed correctly. The failure is not just a calibration issue — the models show a systematic bias toward assuming correct execution regardless of what they actually see, and they can be swayed by superficial changes in how a question is worded. This matters because the commercial promise of AI sports coaching and telehealth movement analysis rests on exactly the capability these models currently lack.
Catalyst
VLMs capable of processing long video clips at reasonable cost only became widely available in 2024-2025, making this kind of systematic benchmark evaluation newly feasible. The paper arrives as several startups and sports-tech incumbents are actively marketing VLM-based coaching products, making an honest capability audit timely and commercially relevant. No prior work had tested this specific failure mode — fine-grained movement quality discrimination — across multiple activity domains and prompting strategies at once.
What's New
Prior Action Quality Assessment (AQA) research relied on task-specific supervised models trained on labeled video datasets, such as convolutional or transformer networks fine-tuned on diving or gymnastics footage. Those models achieved reasonable correlation with human judges but required extensive labeled data and couldn't generalize across sports. This paper instead evaluates general-purpose VLMs zero-shot (no task-specific training), testing whether their broad visual and linguistic knowledge can substitute for specialized training — and finds that it largely cannot, even with advanced prompting tricks like skeleton overlays, chain-of-thought reasoning, or in-context examples.
The Counter
This paper tests VLMs in zero-shot or few-shot settings — without any fine-tuning on AQA-specific data. That's a tough benchmark, but it's arguably the wrong one: no serious deployment would skip domain-specific fine-tuning. The paper's own results show isolated gains from skeleton overlays and in-context learning, which hints that the right training signal, not just prompting tricks, could close much of the gap. The activity domains tested (figure skating, diving, fitness exercises) are visually subtle and even human experts disagree; near-random performance on a hard inter-rater-agreement task is less damning than it sounds. Finally, video understanding in VLMs is advancing rapidly — models tested here may be superseded by architectures with better temporal reasoning within 12 months, potentially invalidating the benchmark before practitioners can act on it.
Longs
- GOOG — Gemini model improvements directly relevant to closing identified gaps
- SYNA — edge inference chips for on-device sports/PT video analysis
- ANET (Arista) — video analytics infrastructure for broadcast sports
- BOTZ (robotics/AI ETF) — broad exposure to embodied AI sensing improvements
- NKE — Nike's athlete data platform would benefit from reliable movement AI
Shorts
- Sports-tech startups currently marketing VLM-based coaching without task-specific fine-tuning — their product claims are directly contradicted
- Telehealth PT platforms that have shipped movement-quality features built on general-purpose vision APIs without specialized validation
- Competitive judging technology vendors positioning VLMs as a near-term replacement for human judges
Enablers (Picks & Shovels)
- MediaPipe and similar skeleton-extraction libraries — the paper tests skeleton overlays as a prompting aid
- AQA benchmark datasets: AQA-7, FineDiving, MIT-Skate — the evaluation's empirical foundation
- OpenAI, Google DeepMind, and Qwen model APIs — the systems actually tested
- Pose estimation research (e.g., ViTPose) — a prerequisite for richer movement representations
Private Watchlist
- Sency (movement analysis AI for fitness and PT)
- HomeCourt (basketball skill tracking app)
- Kitman Labs (sports performance analytics)
- Kaia Health (digital physical therapy using pose estimation)
Resources
The Paper
Action Quality Assessment (AQA) has broad applications in physical therapy, sports coaching, and competitive judging. Although Vision Language Models (VLMs) hold considerable promise for AQA, their actual performance in this domain remains largely uncharacterised. We present a comprehensive evaluation of state-of-the-art VLMs across activity domains (e.g. fitness, figure skating, diving), tasks, representations, and prompting strategies. Baseline results reveal that Gemini 3.1 Pro, Qwen3-VL and InternVL3.5 models perform only marginally above random chance, and although strategies such as incorporation of skeleton information, grounding instructions, reasoning structures and in-context learning lead to isolated gains, none is consistently effective. Analysis of prediction distributions uncovers two systematic biases: a tendency to predict correct execution regardless of visual evidence, and a sensitivity to superficial linguistic framing. Reformulating tasks contrastively to mitigate these biases yields minimal improvement, suggesting that the models' limitations go beyond these biases, pointing to a fundamental difficulty with fine-grained movement quality assessment. Our findings establish a rigorous baseline for future VLM-based AQA research and provide an actionable outline for failure modes requiring mitigation prior to reliable real-world deployment.