A dramatic resignation from inside two of the world’s leading artificial intelligence labs has reignited the debate over whether the race to build increasingly powerful AI systems is moving faster than the safeguards designed to control them.
Jacob Coxon, a 27-year-old British researcher who spent the past three years working on AI pretraining at OpenAI and Anthropic, announced that he is leaving Anthropic — and the AI industry altogether.
His reason was not an ordinary career change.
Coxon said he no longer wants to participate in an industry-wide race to develop increasingly autonomous AI systems that could eventually become capable of improving themselves.
“Neither company is acting responsibly,” Coxon wrote in a public statement. He accused OpenAI and Anthropic of racing toward self-improving superintelligence and “gambling with our lives.”
His departure has drawn particular attention because the warning is coming from someone who worked directly on the technology at the center of the debate.
FROM OPENAI TO ANTHROPIC
Coxon specializes in “pretraining,” one of the fundamental stages involved in developing large artificial intelligence models.
During pretraining, models process enormous amounts of data to develop the basic capabilities that later allow systems such as ChatGPT and Claude to understand language, write software, reason and perform other tasks.
Coxon worked on OpenAI’s technical staff from 2023 until July 2026, with his research including work related to GPT-4o.
He then moved to Anthropic, a company founded by former OpenAI employees and widely known for emphasizing AI safety.
Coxon said Anthropic’s approach to safety was one of the reasons he decided to join the company.
But his experience there ultimately failed to ease his concerns.
According to Coxon, researchers at Anthropic genuinely understand the potential dangers of highly advanced AI, but the company remains trapped in a competitive race in which slowing down could allow another laboratory to move ahead.
“THINGS COULD BE OUT OF CONTROL”
One of Coxon’s most striking warnings concerns how quickly the technology could develop.
Speaking to The Wall Street Journal, Coxon said that under some of the most aggressive scenarios, developments could already be “out of control” by the end of next year.
His concern centers on the possibility of AI systems becoming capable of significantly contributing to the development of even more advanced AI.
Such a system would not merely answer questions or complete tasks assigned by humans. It could potentially help researchers write code, conduct experiments, discover improvements and accelerate the creation of its successors.
Researchers sometimes refer to the theoretical acceleration of this process as recursive self-improvement.
Coxon fears that once AI begins substantially accelerating AI research itself, technological progress could become increasingly difficult for humans to monitor and control.
It is important, however, to distinguish these warnings from established facts. Current systems such as ChatGPT and Claude are not autonomous superintelligences that independently and continuously improve themselves. Coxon is describing a possible future scenario and the risks he believes could accompany it.
THE RACE TOWARD SELF-IMPROVING SUPERINTELLIGENCE
At the center of the controversy is the concept of “self-improving superintelligence.”
Artificial superintelligence generally refers to a hypothetical AI system whose capabilities would exceed those of humans across a broad range of intellectual tasks.
Coxon argues that future systems could become exceptionally capable in areas such as software development, cybersecurity, scientific research and AI development itself.
He warned that sufficiently advanced systems could potentially discover software vulnerabilities, transform industries at extraordinary speed and acquire significant influence or resources in the real world.
Again, these are predictions about potential future systems rather than capabilities demonstrated by today’s consumer AI products.
But Coxon argues that the pace of progress means the industry should address those risks before such systems exist.
“THE PEOPLE BUILDING AI” ARE WORRIED
Coxon also made a striking claim about attitudes inside leading AI laboratories.
He said some of the people developing frontier AI systems genuinely believe the technology could eventually pose an existential threat to humanity.
According to Coxon, executives and senior researchers may use more cautious language when speaking publicly, while privately expressing much stronger concerns about what increasingly capable AI systems could become.
His claims gained additional attention when researchers still working at Anthropic publicly responded.
ANTHROPIC SAFETY RESEARCHER PUTS THE RISK ABOVE 10%
Evan Hubinger, who leads an AI alignment safety team at Anthropic, publicly backed a significant part of Coxon’s warning.
Hubinger said he personally assigns a greater than 10% probability to AI causing human extinction within the next decade.
He also acknowledged a fundamental unresolved challenge facing the industry: ensuring that a future superintelligent system remains aligned with human goals and values.
Hubinger said Anthropic is trying to address the problem but does not yet have a proven plan for solving alignment for superintelligence.
“Alignment” refers broadly to the challenge of ensuring that an AI system continues to behave according to intended human goals and safety constraints even as the system becomes more capable.
Hubinger emphasized that his concern is primarily about future self-improving systems rather than the level of danger posed by today’s models.
His percentage estimate is also a personal risk assessment, not a scientific consensus or a prediction that such an outcome will occur.
CONCERNS ARE SPREADING BEYOND ANTHROPIC
The debate is not limited to Coxon and Anthropic.
Researchers and executives across the AI industry have increasingly called for stronger safeguards as frontier models become more capable.
OpenAI Chief Scientist Jakub Pachocki recently argued that the rapid rise of machine intelligence calls for “extreme caution” and greater coordination between AI companies and governments.
OpenAI CEO Sam Altman has also warned that increasingly powerful AI systems could create serious cybersecurity risks if governments and companies fail to respond quickly.
The unusual situation has created a fundamental dilemma for the AI industry: some of the people building the world’s most powerful AI systems are simultaneously warning that the technology could become extremely dangerous.
THE BIG PROBLEM: CAN ANY COMPANY AFFORD TO SLOW DOWN?
For Coxon, the problem is bigger than the safety policies of OpenAI or Anthropic.
It is the structure of the AI race itself.
If one company voluntarily slows development to conduct additional safety research, another company — whether in the United States, China or elsewhere — could continue advancing and gain the technological lead.
That creates an incentive for every laboratory to keep moving.
Coxon therefore argues that voluntary action by individual companies may not be enough.
Possible solutions could include agreements between leading AI laboratories to limit the pace of development, stronger government oversight or temporary restrictions on developing certain advanced capabilities.
The central challenge is ensuring that companies face similar rules so that prioritizing safety does not automatically mean losing the AI race.
A WARNING FROM INSIDE THE AI INDUSTRY
Coxon’s resignation does not prove that artificial intelligence is about to escape human control.
Nor does it mean that today’s ChatGPT, Claude or other AI models have become autonomous superintelligent systems.
What makes the story significant is where the warning is coming from.
Coxon spent years working directly on frontier AI models at two of the most influential laboratories in the industry.
His concerns are also being echoed, at least in part, by researchers who remain inside those organizations.
That is turning what was once largely a theoretical debate about future artificial intelligence into an increasingly public disagreement among the people actually building the technology.
The question facing the industry is becoming harder to ignore:
Can the world’s leading AI companies slow the race long enough to make increasingly powerful systems safe — or will competitive pressure push the technology forward faster than humanity can learn how to control it?
