Anthropic Researcher Says AI Has Over a 10% Chance of Killing All Humans

Jacob Coxson, a researcher at Anthropic, resigned due to concerns about the irresponsible development of AI by both OpenAI and Anthropic, which he believes poses a significant risk to humanity. He stated that those involved in AI development recognize the potential for it to lead to human extinction by the end of the decade, a risk he argues is unparalleled by any other human activity.

Evan Hubinger, another researcher, echoed these concerns, predicting a greater than 10% chance that AI could result in human extinction within the next ten years. He acknowledged that while the risk from current AI models is low, there is no effective plan in place to address alignment issues for super-intelligent systems, despite Anthropic's efforts.

Coxson warned that the ongoing race in AI development is a reckless gamble, asserting that the industry is not on track to prevent a global race that could have catastrophic consequences. He highlighted that advancements in AI are accelerating, with super-intelligent systems potentially gaining the ability to hack into critical infrastructure and wield real power.

The episode also explored the broader implications of AI, including predictions that it could disrupt a significant portion of entry-level white-collar jobs soon. Bernie Sanders is preparing to introduce legislation aimed at banning superintelligence and pausing AI development, reflecting the growing concerns surrounding the technology's risks.

Discussions on AI regulation revealed a divide, with some arguing that the tech industry is actively lobbying against safety measures. A recent cybersecurity breach related to AI raised alarms about its behavior, underscoring the urgency of addressing these risks.

An unnamed speaker reiterated the claim that AI has over a 10% chance of leading to human extinction within the next decade, emphasizing that this prediction pertains to future scenarios rather than current AI capabilities. This speaker suggested that society must establish policies regarding superintelligence before it becomes a reality.

John Shulman argued for collaboration among industry leaders, specifically OpenAI and Anthropic, to create a pacing proposal for responsible AI development. The conversation around AI risks has intensified with each significant advancement in AI capabilities, highlighting the need for coordinated efforts.

Another speaker noted that while discussions about AI risks, particularly in relation to China, are often viewed as dangerous, they are essential for ensuring future safety. There is an expectation that the current debate on AI risks will experience fluctuations, with periods of heightened concern followed by quieter phases.

The need for genuine coordination among AI industry leaders was emphasized, along with the observation that media coverage tends to amplify extreme viewpoints, complicating efforts to build consensus on AI safety and regulation.