OpenAI safety leader quits, warning AI company's culture is 'broken'

David Robinson, a safety leader at OpenAI, has quit the company, warning that its culture was broken and that AI firms were not “being nearly careful enough” about developing the technology. Robinson led the writing of safety reports that accompanied the ChatGPT developer’s product releases, and explained his resignation in an essay headlined “I quit OpenAI because its culture is broken”. He wrote that a cultural overhaul was needed at cutting-edge AI firms, and said incidents such as a “swarm” of OpenAI agents — AI programmes operating autonomously without human oversight — attacking the AI startup Hugging Face were “typical of the industry, given the speed and flexibility with which people operate”. Writing in The Atlantic, Robinson said he agreed with other recently departed staff that companies building the technology “aren’t being nearly careful enough”, but argued that the focus needed to go deeper than specific rules or new laws: “We need to talk about culture.” Referring to OpenAI’s pace of development, he wrote: “As the company sprints from one launch to the next, it is failing to achieve the level of care that I believe is needed.”

OpenAI has shown signs of caution in recent weeks following the Hugging Face incident and the revelation that it has notified more than 100 organisations about rogue agent activity. This week it announced it was scrapping the release of a next-generation AI model after researchers raised safety concerns during internal testing, and it has also paused training of its most advanced models.

Geoffrey Irving, who worked at OpenAI and DeepMind before becoming chief scientist of Resolution, also joined the warnings on AI on Saturday. Writing in Time, he said: “Recent warnings about the potential destructive power of AI are understating the severity of the situation.” He added: “I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome.”

Robinson’s essay also follows the resignation of Jacob Coxon, a researcher at OpenAI rival Anthropic, who quit the Claude chatbot developer last month. He warned AI “could kill us all by the end of the decade” — and was followed by Anthropic warning there was a more than 10% chance AI would wipe out humanity within the next decade. Critics of such warnings have cautioned, however, that they are unscientific because they cannot be verified or falsified.

Robinson wrote that Silicon Valley lacked an awareness of “how to handle dangerous technology” and “what it means to care for people”. Warning that OpenAI had “unimpeded optimism” about solving problems as they arose, he wrote that this internal culture meant safety failures would only grow as systems become more capable. “Imagine ‘rogue’ agents that work like teams of hackers (for example, holding hospital computer systems for ransom) but never need to sleep,” wrote Robinson.