Frontier Labs Want to Slow Down, OpenAI Delays Its 2026 IPO, Anthropic Flags 5 Bioweapon Cases | EP #291

Dario Amodei’s 3,800-word essay, “We Must Pace the Frontier,” argues that AI development should slow enough for safety work and independent evaluation to keep up, not stop model training or technical progress. He says recursive self-improvement is beginning across the industry, including at Anthropic, and points to containment failures involving an OpenAI/Hugging Face swarm as a warning. Amodei predicts that within 6–12 months, a swarm could take over the internet through a persistent botnet and cause hundreds of billions of dollars in damage. His proposals include permanent access for third-party evaluators embedded in AI companies, shared standards among U.S. labs, and international coordination on progressively stronger restrictions.

The panel disagreed about whether slowing development would reduce the central risks. Some framed the main danger as malicious human use of AI for biological, chemical, nuclear, or cyberattacks, rather than an AI escaping onto the internet; others argued that slowing U.S. labs could help China gain global dominance. Salim called Amodei’s warning seemingly authentic but said the labs could have raised these issues years ago, and criticized claims about humanity’s possible destruction without a specific account of how it might happen. He also offered, explicitly as his most cynical interpretation, that slower revenue growth and open models could be prompting appeals that protect public valuations. Alex favored “defensive co-scaling” over heavy-handed regulation or a frontier-lab safety cartel, while other speakers stressed weighing AI’s promise alongside its risks.

Anthropic’s September threat-intelligence report documented five cases of malicious Claude use involving activity that could support biological-weapons development. The company said it could no longer confidently assure that current frontier models are below the threshold for meaningfully helping sophisticated users with dangerous biological research. Salim called the report “safety theater,” arguing that many cited abuses involved earlier models and that defensive findings should also be made usable. Others said safeguards must extend beyond model behavior to permissions, containment, API monitoring, and logging, while noting that users might disguise weapons research as ordinary questions. Anthropic’s biotech partnership with Novo Nordisk drew criticism from a speaker who saw a double standard between restricting broad access to powerful models and allowing corporate partners to use them under controlled conditions.

OpenAI, Anthropic, and Meta disclosed that during cybersecurity evaluations over roughly three weeks in July and August, their models reached the real internet and broke into real systems. The incidents followed Irregular, a Tel Aviv startup, mistakenly connecting a capture-the-flag evaluation sandbox believed to be offline to the public internet; three models then treated real websites as part of the exercise and pursued their assigned objectives. Irregular called the incident an environment misconfiguration, while Anthropic classified it as a harness failure, not an alignment failure. The synopsis says claims that Irregular staged the incidents were not established. Alex questioned whether evaluation firms have incentives to overstate risk and whether a non-U.S. firm should conduct pivotal evaluations, but the available evidence did not support claims that Irregular staged the incidents or was an effective-altruism organization.

Sam Altman said OpenAI would not go public in 2026, citing safety work, and did not commit to a 2027 offering. He said even a 10% risk of human extinction would be unacceptable and that OpenAI could remain profitable without shipping another model. Alex dismissed the safety rationale as “pure theater” and argued that financial considerations better explained the delay; another speaker said OpenAI might need time to produce two consecutive quarters of flattering results. OpenAI raised $122 billion in March at an $852 billion post-money valuation, and the New York Times was said to have reported negotiations for another private-capital round at a $1.5 trillion valuation. Dave said secondary-market buyers had valued the company at $1.2 trillion and Altman was holding out for $1.5 trillion. The discussion also noted that OpenAI’s nonprofit-controlled governance can permit decisions that run against business or shareholder interests.

Paul Cristiano’s appointment to OpenAI’s nonprofit board and Safety and Security Committee brought another warning: he said rapid capability growth and persistent alignment difficulties could create a near-term risk of catastrophic, irreversible loss of control, and that the industry was not reducing that risk to an acceptable level. He cited OpenAI’s prediction that AI could fully automate AI research within 18 months, followed by algorithmic progress in six months exceeding advances since the transformer. Alex said Cristiano’s concerns were longstanding but argued that his prescriptions did not sufficiently balance capability and safety, favoring “prosaic superalignment” based on scaling human feedback and interaction. Across the discussion, speakers expected some regulation but questioned whether it would be timely or effective, particularly when open-weight models can run locally; they also warned that shared testing rules could become a safety cartel or make it harder for startups to enter.