AI researcher Jacob Coxon has left Anthropic, the company behind Claude, and decided to quit the industry, citing serious concerns about the safety of the technology. He said that companies like Anthropic and OpenAI are moving too fast in the race to build more powerful AI systems and that could eventually kill humans by the end of the decade.
Coxon, who’s also worked with OpenAI in the past, said many executives and senior AI researchers believe the same, even though they do not speak up openly. The 27-year-old received backing from his former colleague and current Anthropic employee Evan Hubinger.
"Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve…. for superintelligence and are not clearly on track to," Hubinger said as he shared Coxon's post on X.
What’s superintelligence and why are they concerned?
Superintelligence refers to a hypothetical AI system whose abilities could surpass gifted human minds. Coxon said these companies are aware that advanced AI could potentially pose an existential risk to humans, but are still competing to achieve superintelligence first.
"Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing," Coxon wrote.
"The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger," he added.
What happens next?
Coxon believes government intervention or a coordinated slowdown among tech companies is needed before advanced AI systems are developed.
Anthropic CEO Dario Amodei had himself said in July that the US government must support a coordinated slowdown. However, in February this year, the company diluted its own safety commitments by removing a pledge to halt development if risks became unmanageable. Anthropic said that even if it paused development, its less cautious competitors would not.
Meanwhile, AI companies continue to invest heavily in their advanced models without much government oversight.
