Phone AI Benchmarks Flawed: New Study Exposes CLI, API Gap

New research reveals that benchmarks for AI agents operating smartphones have been measuring only the easiest part of their job. A new framework called PhoneHarness shows that current large language models struggle with more complex tasks. Previous tests focused mainly on graphical user interface, or GUI, performance. However, real mobile workflows also involve shell commands and programmatic API calls. PhoneHarness evaluates agents across all three areas, running on actual Android environments. This timing is important because major AI labs are developing phone-use agents, with some even planning AI-centric smartphones. Their commercial pitches have relied heavily on benchmark performance. If these benchmarks were incomplete, the industry's self-assessment has been flawed. What's interesting is that GUI scores do not predict how well agents perform with shell commands or API calls. The problem isn't reasoning, but that different tasks require different output formats and training data. This matters because it highlights a significant gap in current AI agent capabilities for real-world phone automation.