Former OpenAI safety employee warns company culture is failing to keep pace with AI risks

A former OpenAI safety employee has criticised the company’s organisational culture, arguing that its emphasis on rapid development and iterative deployment could increase the risk of safety failures as AI systems become more capable.

Writing in The Atlantic, David Robinson, who spent three-and-a-half years at OpenAI and helped draft its preparedness framework, said AI companies were “not being nearly careful enough” and needed to look beyond individual safety rules towards the culture shaping how advanced systems are developed.

A culture built around speed

Robinson said OpenAI’s pace of development was making it difficult to achieve the level of caution he believes is necessary for frontier AI systems.

“As the company sprints from one launch to the next, it is failing to achieve the level of care that I believe is needed.”

He argued that the industry’s reliance on “iterative deployment”, where systems are released and safeguards strengthened as problems emerge, is no longer sufficient for increasingly powerful AI.

“I agree with other recently departed staff that the companies building this technology aren’t being nearly careful enough. But I believe that we need to look deeper than specific rules or new laws. We need to talk about culture.”

Robinson said Silicon Valley needed a stronger understanding of how to manage dangerous technologies and what it means to build systems with greater care for people.

Safety expertise needs a bigger role

Robinson called for frontier AI companies to draw more heavily on safety practices from industries such as nuclear power and aviation, where multiple layers of safeguards are designed to limit the impact of human error.

“Given today’s risks, frontier labs need to run like nuclear power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster.”

He also called for “new science” to ensure that increasingly powerful AI systems can be controlled when operating autonomously.

His concerns come amid wider scrutiny of AI companies following incidents involving autonomous agents and failures in safety controls. 

Robinson cited an incident involving a “swarm” of OpenAI agents operating without human oversight as an example of the risks created by a fast-moving development culture.

OpenAI defends its safety approach

OpenAI has indicated that it has taken steps to strengthen its safety practices, including pausing training and holding back models when concerns emerge.

“We’re making sure our models don’t become more capable than we can safely manage and secure, and we pause training or hold back models when we need to slow down,” an OpenAI spokesperson said.

The company also said it was continuing to strengthen its safety and security practices to address both current risks and those that could emerge from future AI breakthroughs.

Robinson’s departure adds to growing debate within the AI industry over whether organisational culture, rather than technology or regulation alone, is becoming a critical part of the safety challenge.