LLMs Struggle with Cyberattack Chain Reconstruction: DiagChain

A new diagnostic benchmark called DiagChain shows that large language model agents still have a long way to go in reconstructing cyberattack chains. Security researchers have hoped these models could speed up this tedious incident response task. DiagChain evaluates how and why LLM agents fail when trying to piece together an attacker's actions from system telemetry. It goes beyond simple accuracy scores to pinpoint where reasoning breaks down. The benchmark includes a suite of 69 scenarios, called MAIN-69. These scenarios cover various operating systems, different levels of evidence noise, and varying chain lengths. What's interesting is that across six different LLMs tested, the best configuration only succeeded on 39.6% of 849 reference steps. Smaller models struggle to use retrieved evidence, while larger models mostly struggle with ordering that evidence correctly. This matters because cybersecurity teams need to know if an AI agent is trustworthy enough to help with real intrusions.