Atomicwork & New Measure Open Source ITSM AI Benchmark

Atomicwork and New Measure have open-sourced ITSMBench. This new AI benchmark helps developers and businesses compare AI models for IT service management tasks. Here's the thing: ITSMBench is entirely open source. This means researchers and companies can reproduce results, inspect the benchmark, and build upon it. The benchmark recreates enterprise service desks using 42 mocked software systems, nearly 1,800 database tables, and over 2,000 REST endpoints. It covers 89 service-desk tasks, including identity management, networking, and security operations. Initial results show differences between models. For instance, Grok 4.5 leads in tool discovery at 83.0%, but it's near the bottom for task completion. Opus 5 leads in execution at 63.5% once tools are available, despite a lower discovery rate. No single model solved all 89 tasks. The bottom line: blindly trusting an AI agent right now could lead to security issues, according to Atomicwork's CEO. This highlights the need for enterprises to optimize for accuracy, cost, and resolution time when using AI for service desks.