KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation
A new benchmark reveals that today's best AI agents fail more than half the time when they must infer what a user actually wants rather than follow explicit instructions.
The Thesis
Mobile AI agents — software that operates smartphone apps on your behalf — are becoming a real product category, but nobody has had a good way to test whether they can handle the messy reality of personal preferences and uninvited help. KnowU-Bench fills that gap by building a rigorous test environment on an Android emulator, hiding the user's preference profile from the agent and forcing it to figure out what you want through conversation. The result is a sobering finding: even frontier models like Anthropic's Claude Sonnet 4.6 fall below 50% success on tasks requiring preference inference or judgment about when to act without being asked. The bottleneck is not navigating app interfaces — agents are already decent at that. The real gap is knowing when to ask, when to act, and when to stay out of the way, which is exactly what a trustworthy personal assistant requires. Until this gap closes, fully autonomous mobile agents will remain a liability risk more than a product.
Catalyst
LLM-driven agents capable of operating smartphone GUIs (graphical user interfaces) have matured enough in the past year that companies are actively shipping mobile agent products, making the absence of rigorous evaluation frameworks an urgent problem rather than a theoretical one. Android emulation tooling has also reached a point where reproducible, stateful benchmark environments are tractable to build and share. The commercial pressure from Apple Intelligence, Google Gemini integration, and a wave of agent startups has concentrated industry attention on exactly the capabilities this benchmark probes.
What's New
Prior mobile agent benchmarks such as AndroidWorld and MobileAgentBench evaluated agents on explicit, fully specified tasks — the user tells the agent exactly what to do and the agent either does it or doesn't. Some preference-related benchmarks treated the user profile as static context handed directly to the model, which is more like open-book recall than real inference. KnowU-Bench withholds the user profile entirely, exposes only behavioral logs, and scores agents on their ability to elicit missing information through multi-turn dialogue and to decide proactively whether an intervention is warranted — a fundamentally harder and more realistic test.
The Counter
KnowU-Bench is a benchmark paper, not a deployed system — it measures a capability gap but does not demonstrate how to close it, and capability gaps on benchmarks have a way of closing faster than the benchmark authors expect once the specific test becomes public. The 42 general, 86 personalized, and 64 proactive tasks is a reasonable but still narrow sample; task coverage and Android-only scope may not generalize to iOS-dominated Western markets or enterprise desktop environments. The LLM-as-a-Judge scoring component introduces its own reliability questions: the judge model's own preferences and failure modes could systematically bias scores in ways the paper does not fully characterize. Most importantly, the finding that agents fall below 50% on hard preference tasks is striking, but it may reflect prompt engineering choices rather than fundamental architectural limits — the same models might perform much better with targeted fine-tuning or retrieval-augmented preference memory, neither of which was tested here.
Longs
- GOOGL — Android platform owner; agent capability improvements flow through its ecosystem first
- AAPL — competing Apple Intelligence agent stack; benchmark pressure accelerates investment
- SAMSF (Samsung Electronics OTC) — device maker with Bixby agent roadmap and strong Android market share
- BBAI — small-cap AI agent infrastructure play with enterprise automation exposure
- BOTZ (Global Robotics & Automation ETF) — broad exposure to AI agent and automation themes
Shorts
- Teams selling mobile agent products on explicit-task benchmarks alone — KnowU-Bench exposes that those scores don't predict real-world personalization performance
- Benchmark incumbents like AndroidWorld — if KnowU-Bench becomes the standard, prior leaderboard positions become less meaningful for enterprise procurement decisions
- Agent startups that have marketed autonomy without investing in preference elicitation and consent mechanics — this paper quantifies exactly where they fall short
Enablers (Picks & Shovels)
- Android Open Source Project (AOSP) emulation tooling — makes reproducible stateful environments possible
- Anthropic API and OpenAI API — frontier models used as both agents under test and LLM-as-a-Judge evaluators
- LLM-as-a-Judge evaluation frameworks (e.g., MT-Bench methodology) — hybrid scoring depends on this
- Structured user profile datasets for preference simulation — the benchmark's user simulator depends on realistic profile data
Private Watchlist
- Anthropic (private) — Claude Sonnet is directly benchmarked; pressure to improve preference inference
- Adept AI (private) — building general computer-use agents, directly in scope
- Rabbit (private) — consumer mobile agent hardware/software, exactly the product this benchmark targets
- MultiOn (private) — web and mobile agent automation startup
- Induced AI (private) — browser and mobile automation agent infrastructure
Resources
The Paper
Personalized mobile agents that infer user preferences and calibrate proactive assistance hold great promise as everyday digital assistants, yet existing benchmarks fail to capture what this requires. Prior work evaluates preference recovery from static histories or intent prediction from fixed contexts. Neither tests whether an agent can elicit missing preferences through interaction, nor whether it can decide when to intervene, seek consent, or remain silent in a live GUI environment. We introduce KnowU-Bench, an online benchmark for personalized mobile agents built on a reproducible Android emulation environment, covering 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. Unlike prior work that treats user preferences as static context, KnowU-Bench hides the user profile from the agent and exposes only behavioral logs, forcing genuine preference inference rather than context lookup. To support multi-turn preference elicitation, it instantiates an LLM-driven user simulator grounded in structured profiles, enabling realistic clarification dialogues and proactive consent handling. Beyond personalization, KnowU-Bench provides comprehensive evaluation of the complete proactive decision chain, including grounded GUI execution, consent negotiation, and post-rejection restraint, evaluated through a hybrid protocol combining rule-based verification with LLM-as-a-Judge scoring. Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions requiring user preference inference or intervention calibration, even for frontier models like Claude Sonnet 4.6. The core bottlenecks are not GUI navigation but preference acquisition and intervention calibration, exposing a fundamental gap between competent interface operation and trustworthy personal assistance.