CoArena: Crowdsourced AI Agent Benchmark for Real-World Tasks

CoArena is developing a crowdsourced benchmark for computer-use AI agents. This platform, previously known as Coasty, evaluates agents on real-world tasks using live trials. Here's the thing: It contrasts polished demonstrations with unscripted attempts by agents to complete the same task. CoArena's video mentions 10,000 agent runs and 17,000 human verdicts. The company aims to assess if agents can reliably complete practical computer workflows, not just produce convincing demos. What's interesting is how it works: People judge the results of agents performing live tasks. The website highlights blind human judgment and published numbers with visible rules. This approach reveals failures that static benchmarks might miss. The bottom line: Computer-use agents need to handle multi-step actions across various interfaces. A benchmark based on independent runs and human outcomes offers a direct view of their reliability. This matters because it helps developers and teams decide which computer-use systems are truly ready for broader deployment.