AI Incidents Rise: New Playbook Needed for Model Risks

Old security playbooks are not equipped to handle AI glitches. Companies need a new strategy to address model errors and legal risks. Seventy-one percent of organizations report that AI has access to core business systems. However, only 16% effectively govern this access, according to a recent report. AI incidents increased by 56.4% from 2023 to 2024, with 233 documented cases. These incidents fall into two main categories: failures caused by the model itself and failures caused by human interaction. Model-originated failures include issues like degradation, bias, or hallucinations. For example, the Epic Sepsis Model missed two-thirds of actual sepsis cases. Externally induced failures involve adversarial attacks or data poisoning, such as Tesla's Autopilot phantom braking cases. A hybrid case carries significant legal exposure. Air Canada was held liable when its chatbot invented a bereavement fare policy. In another instance, a US federal court allowed a case to proceed where an AI hiring platform was considered directly liable. These incidents often become legal issues, even if they don't appear to be security incidents. The bottom line is that organizations must update their incident response plans to account for the unique challenges posed by AI.