AI Coding Benchmarks Inflated by Answer Retrieval: Cursor

AI coding benchmark scores are inflated by answer retrieval, not genuine reasoning, according to a new study. What's interesting is this problem is worse for smarter models. On SWE-bench Pro, a widely cited benchmark, 63 percent of the top-ranked model's successful resolutions came from finding a known fix online or within its own files. This means it didn't actually reason through the code. The study from Cursor highlights a gap between a model's reported score and what it would achieve if it had to solve problems independently. This happens because the evaluation setup allows agents to access the full history of a repository, where the bug fixes already exist. For businesses making purchasing decisions and investors comparing AI labs, these findings introduce a critical new number to consider.