ClawBench: Can AI Agents Complete Everyday Online Tasks?
A new benchmark tests AI agents on real websites for real tasks — and finds even the best models complete barely a third of them.
The Thesis
ClawBench is an evaluation framework that tests AI agents on 153 everyday tasks — booking appointments, filling forms, submitting job applications — across 144 live, production websites. Unlike lab-style benchmarks that run on frozen or simulated web pages, ClawBench operates on the actual internet, which means agents face real login flows, dynamic content, and multi-step workflows that change over time. The headline finding is sobering: Claude Sonnet 4.6, one of the strongest available models, completes only 33.3% of tasks. That gap matters because it tells investors and builders where the real capability ceiling is, not where it looks on cleaner, easier benchmarks. The catch is that a benchmark is not a product — high scores here would be necessary but not sufficient for a deployable personal-assistant agent.
Catalyst
Frontier AI models have recently crossed thresholds in web browsing and computer-use capability — Anthropic's computer-use API, OpenAI's Operator, and Google's Project Mariner all launched or matured in the past twelve months — making practical web-agent evaluation both newly relevant and newly possible. At the same time, prior benchmarks such as WebArena and Mind2Web relied on static snapshots or sandboxed replicas of websites, which systematically underestimated real-world difficulty. The authors' lightweight 'interception layer' — a technique that blocks only the final form submission so that tasks can be run safely on live sites — is a relatively new methodological tool that makes live-web evaluation tractable without causing real-world side effects.
What's New
Earlier web-agent benchmarks like WebArena (a sandboxed replica of sites such as Reddit and GitLab) and Mind2Web (a static HTML dataset) evaluated agents on frozen or simulated environments. Those setups are reproducible and safe, but they strip out the variability, pop-ups, CAPTCHA flows, and dynamic state that make real websites hard. ClawBench runs directly on production websites across 15 task categories, preserving full real-world complexity. The authors claim this makes ClawBench meaningfully harder and more predictive of real deployment performance than offline alternatives.
The Counter
A 33% completion rate sounds like a damning indictment of current AI agents — but it also means this benchmark may mostly be measuring website friction, not model intelligence. Production websites have anti-bot measures, inconsistent UI patterns, and aggressive session timeouts that would trip up a human using an unfamiliar site too. The paper evaluates only 153 tasks across 144 platforms, which is a thin sample; a different draw of tasks or platforms could shift scores substantially. There is also a reproducibility problem baked into the design: because tasks run on live websites, the benchmark is a moving target — a site redesign or new CAPTCHA can silently change scores between runs. Finally, the benchmark measures task completion rate as a binary, which may not reflect partial progress that is still commercially valuable. Agents that get 80% of a form filled correctly before failing are far more useful than a 0% score implies.
Longs
- ANTHROPIC (private) — Claude is the top-performing model in the benchmark
- MSFT — OpenAI Operator and Copilot agent roadmap directly implicated
- GOOGL — Project Mariner and Gemini agent stack relevant
- BOTZ (robotics/automation ETF) — broad exposure to software automation theme
- UPWK — Upwork, a platform whose freelance workflows could be disrupted or automated by capable agents
Shorts
- Zapier and Make.com — their value proposition (connecting apps via pre-built integrations) shrinks if general-purpose web agents can navigate any site without custom connectors
- UiPath and Automation Anywhere — RPA (robotic process automation) vendors whose rule-based bots are more brittle and expensive to maintain than LLM-driven agents
- Mechanical Turk-style human microtask platforms — routine form-filling and data-entry tasks are exactly what ClawBench measures
Enablers (Picks & Shovels)
- Playwright and Puppeteer — open-source browser automation libraries that underpin web-agent execution
- Anthropic computer-use API — exposes screen-level control that agents need for live-web tasks
- BrowserBase and similar headless-browser infrastructure providers — supply scalable sandboxed browsers for agent evaluation and deployment
- Common Crawl and live-web scraping infrastructure — needed to keep benchmark tasks current as websites change
Private Watchlist
- Induced AI — browser-automation agent startup
- Convergence AI — personal AI agent focused on task completion
- MultiOn — web-agent platform directly in this benchmark's target domain
- Skyvern — open-source web-automation agent using LLMs and vision
Resources
The Paper
AI agents may be able to automate your inbox, but can they automate other routine aspects of your life? Everyday online tasks offer a realistic yet unsolved testbed for evaluating the next generation of AI agents. To this end, we introduce ClawBench, an evaluation framework of 153 simple tasks that people need to accomplish regularly in their lives and work, spanning 144 live platforms across 15 categories, from completing purchases and booking appointments to submitting job applications. These tasks require demanding capabilities beyond existing benchmarks, such as obtaining relevant information from user-provided documents, navigating multi-step workflows across diverse platforms, and write-heavy operations like filling in many detailed forms correctly. Unlike existing benchmarks that evaluate agents in offline sandboxes with static pages, ClawBench operates on production websites, preserving the full complexity, dynamic nature, and challenges of real-world web interaction. A lightweight interception layer captures and blocks only the final submission request, ensuring safe evaluation without real-world side effects. Our evaluations of 7 frontier models show that both proprietary and open-source models can complete only a small portion of these tasks. For example, Claude Sonnet 4.6 achieves only 33.3%. Progress on ClawBench brings us closer to AI agents that can function as reliable general-purpose assistants.