AI Code Oversight: Security Risks in Frontier AI Labs

AI agents are increasingly writing code for frontier AI labs, with light human oversight. This raises concerns about security risks. Researchers at the University of Oxford and SaferAI analyzed these risks, looking at the people reviewing the code, the monitoring pipelines, and the policies in place. They used safety methods from aviation and nuclear power to assess a generic developer, based on public information from companies like Anthropic and OpenAI. One key finding is that several crucial control actions, like pausing a model or assessing catastrophic risk, appear in safety frameworks without a named owner in the public record. While some of these are legal duties, like those specified in California's SB 53, the lack of clear ownership creates a gap. There's also a gap in deployment control. A role exists to approve temporary model use, but the public record doesn't show a role for pulling that deployment back. Furthermore, monitoring of internal coding agents often happens offline. For example, OpenAI's review arrives about 30 minutes after a session, and Anthropic's risk scores can appear after code has already been reviewed or merged. This means monitoring can lag behind the model's actions, especially when new models are introduced. The bottom line: as AI writes more of its own code, clear oversight and timely monitoring are critical for security.