The AI industry has taken a doomer turn. What now?
MIT Technology Review's The Algorithm newsletter reports that Anthropic CEO Dario Amodei posted an essay calling for a brake on the pace of LLM development, citing looming dangers from cyberattacks and bioterrorism to the potential to wreck the economy. OpenAI CEO Sam Altman, Google DeepMind chairman Demis Hassabis, and SpaceXAI CEO Elon Musk all voiced support; Musk wrote on X, "Dario is right."
The agreement is surreal given that just months ago Musk and Altman sat in court attacking each other's reputations in a failed lawsuit Musk brought against his former OpenAI colleague, ostensibly about whether Altman was a trustworthy steward of such dangerous technology. Amodei's rift with OpenAI is even deeper: Anthropic was founded in 2021 because Amodei didn't think Altman took the risks of the technology seriously enough, and Anthropic and OpenAI have competed in a winner-takes-all race ever since. Hassabis has stayed out of the drama, but his company remains a rival.
Now the top labs' public messaging has taken a doomer turn, with a shared message that the latest generation of LLMs aren't safe and everyone needs to figure out what to do. The article says it is easy to be cynical: it is not clear what any of them mean by a slowdown or how it would work, and these companies care a lot about how they come across. With trillion-dollar IPOs in their sights, OpenAI and Anthropic need to reassure investors they're the grown-ups in the room while also hinting at the power of the monsters they have created—and intend to tame; calling for a slowdown does both.
Yet the vibe at the top of these firms really does appear to have shifted. Amodei's post landed six days after OpenAI published an essay by chief scientist Jakub Pachocki, who also laid out why he is concerned about what will happen if LLM development continues unchecked. Pachocki is worried that OpenAI's ability to build powerful models now far outstrips its ability to monitor and control them.
Amodei and Pachocki each cite the July cyberattack against AI firm Hugging Face by a swarm of OpenAI's agents—a hack OpenAI did not even realize had taken place until days after it was all over—as a wake-up call. But their exact position is hard to pin down: Pachocki both calls for a slowdown and highlights an urgent need to stay ahead, writing, "The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI."
As Pachocki frames it, AI firms are locked in a literal arms race: slowing down is good, winning is better.