Artificial intelligence company Anthropic on Thursday proposed a global pause in building the most powerful AI systems as the latest models begin to show signs they could escape human control.
The San Francisco-based company, which makes the Claude family of AI models, said in a report that a worldwide slowdown in cutting-edge AI development “would probably be a good thing” – but warned that if just one company stopped, rivals would simply drive on.
“We believe it would be good for the world to have the ability to slow or temporarily halt cross-border AI development to allow societal structures and adaptation research to keep pace with technological advances,” it said.
Making a real break work would mean multiple major AI companies in multiple countries — notably the U.S. and China — all agreeing to stop at the same time, under rules that everyone could actually verify, Anthropic said.
“Without a global coordination mechanism, companies and governments will have to make tough decisions about security while under competitive and geopolitical pressure,” it said.
The company has faced backlash from others in the industry — and White House officials — who say its focus on worst-case scenarios overstates the risks and amounts to a strategy to slow down rivals under the guise of security concerns.
Still, the White House has recognized the strength of the company’s Mythos model — which has not been made available to the general public because of its cybersecurity capabilities and is currently only deployed to a small number of controlled organizations.
The proposal would face an uphill battle in Washington and Silicon Valley, where US officials and tech executives have repeatedly argued that any slowdown in AI development risks giving China a decisive strategic advantage in what many see as the defining technology race of the century.
However, US President Donald Trump said he discussed the possibility of cooperating with China on AI security issues during his recent visit to Beijing.
Trump also signed an executive order this week giving the government 30 days to conduct a preliminary review of the most powerful US AI models before their release.
‘The human role is narrowing’
Anthropic likened the problem to nuclear arms control treaties – but said it would be even harder to get a handle on, as AI training is far easier to hide than a missile silo, and the temptation to quietly proceed would be huge.
The company said it plans to bring together government officials, scientists, advocacy groups and competing AI firms in the coming months to figure out how such a system could work.
The call for coordination comes alongside internal data showing that AI is already dramatically accelerating the development of AI itself, Anthropic said.
That acceleration creates a feedback loop that Anthropic warned could eventually lead to what researchers call “recursive self-improvement.”
That’s the idea of an AI system that becomes able to essentially teach itself to become smarter without much human help.
“We are not there yet, and recursive self-improvement is not inevitable,” the report said, adding that it could come sooner than most governments and institutions are ready for.
“The evidence suggests that the human role is narrowing at each stage of the AI development process,” the company said.
