Researcher Jacob Coxon submitted his resignation from the artificial intelligence company Anthropic due to his fears that, along with its competitors, the company is participating in a race to develop systems that it will ultimately be unable to control, Researcher Jacob Coxon submitted his resignation.
Coxon, who had specialized in training new artificial intelligence models by processing massive amounts of data, announced his decision in a series of posts on X, explaining that he is leaving Anthropic because he does not want to participate in the industry’s widespread effort to create AI systems capable of improving themselves.
ΣΧΕΤΙΚΑ ΑΡΘΡΑ
Canadian Anne Carson Wins the Nobel Prize in Literature
A Look Back at Apollon’s International Soccer Players
As he writes, he fears that such systems could spiral out of control and threaten humanity.
The 27-year-old Briton, who studied mathematics, noted that many of his colleagues now use terms such as “zero hour” and “endgame” to describe the path toward creating self-improving models.
“We’re heading toward some of the most extreme scenarios, and by the end of next year, things may already be out of control,” Coxon estimates, adding that when companies compete with one another while also facing competition from China, “compromises on security are inevitable.”
Coxon noted that earlier this year he left OpenAI to join Anthropic, as the latter is known for its emphasis on the safety of its models. However, while he believes the company’s efforts are sincere, he now believes that no company can responsibly develop artificial intelligence that surpasses humans across a broad range of tasks—what is often referred to as Artificial General Intelligence (AGI)—without government intervention or a coordinated slowdown by the industry.
The 27-year-old Briton cites in his reasoning recent instances in which models from OpenAI and Anthropic were compromised or operated in collaborative “swarms” of AI agents to emphasize that these systems can adopt malicious goals and attempt to conceal them from humans.
Coxon’s fear is that once these systems begin to improve on their own, they may evolve to the point where they refuse to obey commands.
Coxon’s departure, as the Wall Street Journal notes, follows the resignation, earlier this year, of another researcher working on security issues, who left to pursue poetry, warning that “the world is in danger.”
It should be noted that Coxon was one of more than 1,000 artificial intelligence researchers who recently signed a joint statement calling for international coordination among governments, so that there is the ability to slow down AI development if it becomes necessary to activate a “brake” for systems capable of self-improvement.
Coxon’s departure comes as Anthropic prepares for an initial public offering (IPO), which is expected to be one of the largest in history. The company is seeking a valuation of $2 trillion and has placed particular emphasis on the responsible development of its technology to attract investors.
According to Coxon, Anthropic has a dedicated Slack channel where employees discuss the increasingly powerful capabilities of the company’s models, which reflects just how much influence AI companies now have on the direction the entire industry is taking.
“It’s kind of crazy that all this is happening on the MacBooks of some engineers living in San Francisco, rather than in some isolated desert retreat, as was the case with the Manhattan Project,” Coxon remarked.
Finally, it should be noted that the company’s CEO, Dario Amodei, and other executives have repeatedly warned of the dangers posed by uncontrolled artificial intelligence models and have called on the industry to slow down their development.
